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Preface 


The reason for writing this eHandBook is to provide a short 
easy to read introduction to licensing measurement and 
monitoring systems for licensing researchers and 
administrators, and for regulatory scientists and policymakers. 
It is not intended to be a comprehensive, in-depth analysis of 
licensing measurement. This eHandBook provides only a basic 
introduction to licensing measurement. For those readers who 
are interested in doing a deeper dive into licensing 
measurement, I recommend NARA's - National Association for 
Regulatory Administration's Licensing Curriculum and their 
online courses offered through the University of Southern 


Maine. 


This eHandBook will provide the basics to get the interested 


learner pointed in the direction of learning more about the topic. 


As one will see licensing measurement has its challenges and 
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idiosyncrasies which will need to be addressed by researchers 
and scientists. When I started this journey 50 years ago, I was 
somewhat taken aback by the different data distributions I 
encountered in regulatory science. Back then, regulatory 
science was not well formulated and program monitoring 
related to licensing was more qualitative (case notes) rather than 
quantitative. But I got really interested in public policy and 
macro-systems which seemed to have more and more impact on 
children and their families. This was the beginning of 
governmental rule promulgation, and it was an exciting time to 


be on the cusp of this new research area. 


I learned very quickly that I had to make several adjustments to 


the statistical methods I learned in graduate school to be able to 
analyze licensing data and measure regulatory compliance. 
Several of the theories and methodologies were controversial 
when I proposed them because they went counter to the 


prevailing paradigms at that point in the 1960-70's. However, 
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over time with many replications and validations, the new 
conceptual framework was accepted in the licensing and the 


regulatory science research literatures. 


There are many people to thank over the years and obviously 
this has been a group effort in applying regulatory science to 
early care and education and then expanding it to human 
services and hopefully beyond. What I have found in my most 
recent readings is that regulatory science is being applied in 
many different content silos from the FDA, to economics, to 
banking, and of course within the human services particularly 
adult and child residential services. What appears to be lacking 
is a unifying theory that goes across these disparate content 
areas. That is why I think the introduction of the Regulatory 
Compliance Theory of Diminishing Returns is such an 


important contribution when we _ think about 


licensing/regulatory measurement and monitoring systems. 


The theory has become the foundation for the development of 
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the methodologies and metrics presented in this eHandBook 
such as key indicators, risk assessment, differential monitoring, 
instrument-based program monitoring, integrative quality 
monitoring, skewed data distributions, nominal and ordinal 
measurement scaling, how best to deal with false negatives in 
decision making, and the balancing act between regulatory 


compliance & quality programming. 


As I said earlier this eHandBook needs to be read along with 
the published materials on the Research Institute for Key 


Indicators: RIKI (https://RIKInstitute.com) and the National 


Association for Regulatory Administration: NARA 


(https://www.naralicensing.org/key-indicators) websites. It is 


not intended as a standalone text for licensing measurement. 


Rick Fiene, PhD, Research Psychologist & Regulatory Scientist 
Research Institute for Key Indicators, Penn State University, & 
National Association for Regulatory Administration 


August 2023 
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Chapter 1 


Introduction to Licensing Measurement and 


Systems 


This first chapter provides the learner with an introduction and 
overview to licensing measurement and systems. The 
ehandbook is sponsored by NARA — National Association for 
Regulatory Administration. NARA is the prominent 
international organization dealing with human services 
licensing. This ehandbook is part of the NARA Licensing 
Curriculum which you can find out more about by visiting 
NARA's website (https://www.naralicensing.org/nara- 
licensing-curriculum). NARA also offers a course by the same 
name and this book is the eTextBook for that course; it is 
intended to be used in conjunction with the NARA Licensing 


Measurement course. 


Licensing Measurement & Monitoring Systems 


The NARA course will provide the learner with the major tenets 
of licensing measurement. The learner will discover as they go 
through this book that measurement in licensing is very 
different than other measurement systems found in many of the 
various social and human services. It has some very unique and 
idiosyncratic aspects which will provide us with increasing 
challenges in coming up with specific metrics in determining 


regulatory compliance. 


The field of regulatory science is a very young field. Although 


regulations have been kicking around for well over 100 years, 


the science behind regulations is probably a quarter of this time. 
So, there is not a great deal of empirical evidence to draw upon 
which is discouraging but it is very encouraging and exciting at 
the same time because so much needs to be accomplished in 


establishing regulatory science's theory. 
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A great deal has been written in the past 20 years about 
regulatory science but there has not been a book written about 
measurement. It is hoped that this book will begin that 
discussion. It is also hoped that data driven via regulatory 
science will begin to inform regulatory administration and 


policy more clearly as we move forward. 


This specific chapter will provide the conceptual framework 
and overview to licensing measurement and systems of 


regulatory compliance. It will provide the parameters of the 


book's organization and what will be covered throughout. 


The other chapters to be covered in this book are the following: 


1. Overview/Introduction 
2. Conceptual/Theoretical Framework 
3. Principles of Instrument Design 


4. Regulatory Compliance and Program Quality 


Licensing Measurement & Monitoring Systems Fiene 


5. Coordinated Program Monitoring, Differential Monitoring & 
Integrated Monitoring 
6. What Research Tells Us; What Research Doesn’t Tell Us 


7. Future Directions 


The book is organized into the above 7 chapters. The book is 
short and provides the basics to licensing measurement and 
systems. It is a quick read for regulatory scientists and 
regulatory policy makers as well as licensing administrators. It 
can be read as a standalone text although it was intended as the 
textbook for the NARA Licensing Measurement course, and it is 


recommended to be used with that course. 


The NARA Licensing Measurement course is approximately 45 


hours in length and is organized into anywhere from 7 - 13 


classes. It is equivalent to a three-credit course offered at most 
institutions of higher education. Each class is organized into 
the following: an overview to what will be covered in the 


specific class followed by annotated PowerPoint slides, 
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followed by a series of readings to support the specific 
lecture/PowerPoint slides (I will be referring to these various 
resources throughout this text). For the learner who wants to 
get a thorough grounding in licensing measurement and its 
accompanying program monitoring systems, I _ highly 


recommend them taking the course. 


This book and the course are self-paced and are geared to the 
individual learner. It is totally self-contained meaning that all 
the necessary content is contained with the thirteen classes. If 
a learner just wants to get an overview of what licensing 
measurement is all about, then reading this short ebook will be 
a great start. You can always check out any of the publications 
that are available on the RIKI _ Institute website 


(https://rikinstitute.com/publications/). However, if a learner 


does have a specific question related to this textbook or if they 


are interested in taking the course and would like to get in touch 
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with Dr Fiene, here is his contact information to reach out 


(email address is the best way to contact Dr Fiene): 


Dr Richard Fiene, Research Psychologist & Professor of 
Psychology & Human Development (ret) 
Research Institute for Key Indicators (RIKI) & Penn State 
University 
RFiene@ RIKInstitute.com 
Senior Research Consultant/Senior Regulatory Scientist 


National Association for Regulatory Administration 


RFiene@ NARA Licensing.org 


Also, this ehandbook and the course will draw heavily from 
both the NARA and RIKI websites where many of the 
publications and research reside. Please feel free to go to 


https://RIKInstitute.com/blog/ to download any additional blog 


posts that may be of interest to you. As I said, all the research 


is in the public domain and follows an open science sharing 


arrangement. 
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The links for additional publications (see the references listed 
at the end of this book) for the NARA course are listed 
throughout this book or within the course format in the 
handouts section on the NARA website which you can 
download in their entirety or do it chapter by chapter. All 
course materials will be provided in either the lectures section 


or the handouts section of the class. 


I thought it would be helpful to provide a bit of my background 
which will help the reader to put in context the content of this 


text and the course. This textbook is written by a research 


psychologist who has spent his career in improving childcare 


quality through an early childhood program quality indicator 
model of training, technical assistance, quality rating and 
improvement systems, professional development, mentoring, 
licensing, risk assessment, differential program monitoring, key 


indicators, and accreditation. While content wise, I spent my 
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career in early care and education, I evolved into a regulatory 
scientist because of the various positions I held within 


governmental service and interest area that focused on public 


policy, macro systems, and licensing rules. 


Here are some additional commentaries taken from my NIH Bio 
Sketch that might help to fill in some details related to my 


background: 


Dr Fiene is a retired professor of human development & 
psychology (Penn State University) where he was department 
head and founding director of the Capital Area Early 
Childhood Research and Training Institute. He is presently 
President & Senior Research Psychologist/Regulatory Scientist 


for the Research Institute for Key Indicators. 


Dr Fiene is regarded as a_ leading international 


researcher/scholar on human services licensing measurement 
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and differential monitoring systems. His regulatory compliance 
law of diminishing returns has altered human _ services 
regulatory science and licensing measurement dramatically in 
thinking about how best to monitor and assess licensing rules 


and regulations through targeted and abbreviated inspections. 


His research has led to the following developments: 
identification of herding behavior of two-year old's, national 
early care and education quality indicators, mathematical 
model for determining adult child ratio compliance, solution to 
the trilemma (quality, affordability, and accessibility) in child 
care delivery services, Stepping Stones to Caring for Our 
Children, online coaching as a targeted and individualized 
learning platform, validation framework for early childhood 
licensing systems and quality rating & improvement systems, 
an Early Childhood Program Quality Improvement & Indicator 


Model, Caring for Our Children Basics, and has led to the 


development of statistical techniques for dealing with highly 
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skewed, non-parametric data distributions in human services 


licensing systems. 


Organization of the eHandBook 


In the following chapters, the reader will find how licensing 
measurement is very unique when it comes to regulatory 
science. By knowing these unique characteristics, it will be 
easier to administer and monitor programs governed by 
licensing rules. This can be of benefit to those who administer 
licensing agencies and to those who are asked to conduct 


research on regulatory policies and compliance. 


In Chapter 1, this chapter, provides the basic introduction and 
overview to the ehandbook as well as a short history of 
licensing measurement and a timeline for early care and 
education standards focusing on Caring for Our Children: 


Basics. 
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In Chapter 2, the reader will be introduced to the theoretical and 
conceptual foundations to licensing measurement. The 
regulatory compliance theory of diminishing returns will be 
introduced which has had a tremendous impact on human 
services regulatory science and administration. Paradigm 
alternatives will be suggested that have guided regulatory 


science over the past several decades. 


In Chapter 3, the principles of instrument design will be 
addressed. Obviously without a reliable and valid system of 
measurement it will be simply garbage in and garbage out. This 
is an area when it comes to instrument design that gets the short 
end of the stick many times. The level of measurement will be 


addressed and its impact on the types of statistics selected. 


Also, how best to design data bases will be addressed. 
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In Chapter 4, regulatory compliance and program quality will 
be discussed presenting it on a dichotomous ten-point polemic 
for the regulatory science field to consider. This is a very 
important chapter building off the theory of regulatory 


compliance of diminishing returns introduced in chapter 2. 


In Chapter 5, the essence of program monitoring systems is 
introduced along with differential monitoring and its two major 
methodologies of key indicators and risk assessment. This 
chapter gets us thinking about what a licensing measurement 


will look like administratively. 


Chapter 6 deals with the research literature what we know and 


what still needs to be addressed, the gaps in our regulatory 


science knowledge base. Examples are provided of success 


stories across the USA and internationally. 


Licensing Measurement & Monitoring Systems Fiene 


The last chapter, Chapter 7, provides us with where do we go 
from here with licensing measurement and systems. What are 


the next steps. 


There are a series of technical research notes that form an 
appendix which help to clarify the body of the text and add 
some details. This is followed by a comprehensive reference 
listing and then several figures, charts, graphs, and displays 


which depict various concepts presented in the text. 


A Brief History of Licensing Measurement 


The history of licensing measurement and _ regulatory 


compliance has actually a rather long lineage but is still in its 


infancy in terms of development. In the early stages most 


licensing visits and inspection results were recorded via 
anecdotal records/case records with the licensing staff 


recording their results in more social work note taking. It was a 
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qualitative type of measurement with very little quantitative 
measurement occurring with the exception of basic 
demographics, number of clients, number of caregiving staff, 
etc... This qualitative approach worked very well when there 
were not many programs to be monitored and there were 
sufficient licensing staff to do the monitoring and conduct the 


inspections. 


This all started to change in the 1980’s when Instrument Based 
Program Monitoring (IPM) was introduced and started to be 
adopted by state licensing agencies throughout the United 
States. Just as a footnote, this brief history is pertinent to the 
USA and does not include other countries although the 
Canadian Provinces have followed a similar route as the USA. 
The reason for the introduction of an IPM approach was the 


tremendous increase in early care and education programs in 


the 1960’s and 1970’s. It was difficult for licensing staff to keep 


up with the increased number of programs in their monitoring 
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efforts. There needed to be a more effective and efficient 


methodology to be employed to deal with these increases. 


A very influential paper was written in 1985 and published in 
Child Care Quarterly which introduced IPM along with 
Licensing Key Indicators, Risk Assessment (Weighting), and 
Differential Monitoring (Abbreviated Inspections). This paper 
outlined the various methodologies and their use by a 
consortium of states to test the viability of this new approach to 
licensing measurement, regulatory compliance, and program 
monitoring. Also, the terminology has changed over the 
decades. Back in 1985 weighting was used rather than risk, 
abbreviated inspections were used rather than differential 
monitoring, targeted monitoring, or inferential monitoring. All 
these terms can be used interchangeably as they have been over 
the years, but the first introduction of them back in 1985 utilized 


weighting and abbreviated inspections. 


In the early 1990’s the risk assessment methodology was used 


to develop Stepping Stones to Caring for Our Children, the 
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comprehensive national health and safety standards for early 
care and education (ECE) programs in the USA. This was a 
major development in attempting to develop national voluntary 


standards for child care in the USA. 


It was during this time that two other very significant 
discoveries occurred related to licensing data distributions: 1) 
Licensing data are extremely skewed and do not follow a 
normal curve distribution. This fact has a significant impact on 
the statistics that can be used with the data distributions and 
how data analyses are performed. For example, data 
dichotomization is warranted with licensing data; 2) Regulatory 
compliance data are not linear when compared to program 
quality measures but are more plateaued at the substantial and 
full regulatory compliance levels. The data appear to follow the 
Law of Diminishing Returns as compliance moves from 
substantial to full (100%) regulatory compliance. This finding 


has been replicated in several studies and has been controversial 


because it has led to the issuing of licenses to programs with 
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less than full compliance with all rules/regulations/standards. 
These two discoveries have been very influential in tracking 


developments in licensing measurement since their discoveries. 


In the new century as states began to adopt the various 
methodologies it became necessary to have a standardized 
approach to designing and implementing them. The National 
Association for Regulatory Administration (NARA) took up 
this role and in 2000 produced a chapter on Licensing 
Measurement and Systems which helped to _ guide 
states/provinces in the valid and reliable means for designing 


and implementing these methodologies. In 2002 a very 


important study was conducted by the Assistant Secretary’s 


Office for Planning and Evaluation (ASPE) in which they 
published the Thirteen Indicators of Quality Health and Safety 
and a Parent’s Guide to go along with the research. This 
publication further helped states as they revised their licensing 
and program monitoring systems for doing inspections of early 


care and education facilities based upon the specific indicators 
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identified in this publication. Both publications have been 


distributed widely throughout the licensing world. 


During the first decade of the new century, Stepping Stones for 
Caring for Our Children went through a second edition. This 
publication and the ASPE publications were very useful to 
states as they prepared their Child Care Development Fund 
(CCDF) plans based upon Child Care Development Block 


Grant (CCDBG) funding. 


From 2010 to the present, there have been many major events 


that have helped to shape licensing measurements for the future. 


Caring for Our Children Basics (CFOCB) was published and 
immediately became the default voluntary early care and 
education standards for the ECE field. The CFOCB is a 
combination of the risk assessment and key indicator 
methodologies. Three major publications by the following 
Federal agencies: HHS/ACEF/USDA: Department of Health and 
Human — Services/Administration for Children and 


Families/United States Department of Agriculture, OCC: 


25 


Licensing Measurement & Monitoring Systems Fiene 


Office of Child Care, and ASPE: Assistant Secretary’s Office 


for Planning and Evaluation dealing with licensing and program 
monitoring strategies were published. These publications will 
guide the field of licensing measurement for years to come. The 
Office of Head Start developed and implemented their own 
Head Start Key Indicator (HSKID) methodology. And in 2016, 
CCDBG was reauthorized and differential monitoring was 
included in the legislation being recommended as an approach 


for states to consider. 


Most recently, the Office of Head Start is revising their 
monitoring system that provides a balance between compliance 
and performance. This system revision will go a long way to 
enhancing the balance between regulatory compliance and 
program quality. Also, there has been experimentation with an 
Early Childhood Program Quality Indicator instrument 
combining licensing and quality indicators into a single tool. 
These two developments help with breaking down the silo 


approach to measurement where licensing and quality 


26 


Licensing Measurement & Monitoring Systems Fiene 


initiatives are administered through separate and distinct 
approaches such as licensing versus professional development 
systems versus quality rating and improvement systems. A 
paradigm shift in which an Early Childhood Program Quality 
Improvement and Indicator Model is proposed. The paradigm 
shift should help to make licensing measurement more 


integrated with other quality initiatives. 


The licensing field continues to make refinements to its 
measurement strategies in building a national/international 
regulatory compliance data base. More and more is being 
learned about the nuances and idiosyncrasies of licensing data, 
such as moving from a nominal to an ordinal driven data 
system. For example, NARA and the Research Institute of Key 
Indicators (RIKI) have entered into an exclusive agreement for 
the future development of licensing measurement strategies via 


differential monitoring, key indicators for licensing and 


program quality, and risk assessment approaches. Several 


validation studies have been completed in testing whether the 
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various methodologies work as intended. A significant Office 
of Program Research and Evaluation (OPRE) Research Brief 
which developed a framework for conducting validation studies 
for quality rating and improvement systems has been adapted 


to be used in licensing measurement. 


For additional updates to licensing measurement, please check 
out and follow these RIKINotes Blog posts. There are and will 
be many examples of licensing measurement enhancements. 
Also, although much of the research on licensing measurement 
has been completed in the ECE field, the methodologies, 
models, systems, and approaches can be utilized in any human 


service arena, such as child residential or adult residential 


services. Also, NARA’s chapter in their Licensing Curriculum 


has been developed into a full-blown course, please go to the 
following web page _ for additional information 


(https://www.naralicensing.org/key-indicators). 
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A Timeline of ECE Standards and Program Monitoring in 
the United States: Caring for Our Children Basics—probably 
the best example of a health and safety standards tool 


It all started in and around 1965 when the Federal government 
got into early care and education (ECE) in earnest with Head 
Start and federally funded day care for low-income families. It 
started off slowly but began to pick up momentum with exciting 
studies and research applying principles from developmental 
psychology to policy making. Researchers and policy makers 
wanted to make sure that these new programs were not 
detrimental to young children since our frame of reference were 
children being raised in orphanages and the ultimate outcome 
for children was not positive. Would ECE have the same 


impact? 


Issues around quality, appropriateness of standards, and 
demonstration programs became the focal point of federal 
research funding. The focal point of this essay is on the 


appropriateness of the ECE standards and the resulting 
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monitoring systems that were to become key to the federal 
involvement in early care and education. This essay will be 
organized by the following 50 years neatly broken out by each 
decade to get us from this beginning in 1965 until the 
publication of Caring for Our Children Basics in 2015 by the 
federal government, the Administration for Children and 
Families, U.S. Department of Health and Human Services. A 
look at the 2020 decade with a future note is also appended to 


this essay. 


1970s 


During the 1970’s, the federal government became concerned 


about what were to be the standards for this new national 
program related to federally funded ECE for low-income 
families and their children. Head Start was a separate entity and 
we will revisit Head Start later but our focus for now is on the 
federally funded programs which became known back then 
generically as day care. This nomenclature changed to child 


care and to finally early care and education (ECE) during this 
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50-year history. The initial standards for day care were the 
Federal Interagency Day Care Requirements (FIDCR). A very 
large appropriateness research study led by Abt Associates to 
determine what were the most salient standards and their 
intended impact on children while in day care was conducted 
during this decade. These standards were to be federally 
mandated requirements for any program receiving federal 
funding. This is where group size and adult-child ratios 
standards became such important safeguards and surrogates for 


children’s health and safety in day care programs. 


It also became of interest for the federal government to design 
the monitoring system that would determine compliance with 


the FIDCRs. But it became clear to the original designers of 


this new system that the monitoring of the FIDCR was going to 


be difficult to do across the full USA. So, the question became, 
is there a way to monitor the standards in the most effective and 


efficient manner? This question and the future of the FIDCR 
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were to be altered and put on hold once we moved into the next 


decade. 


1980s 


A change in federal administration and a resulting change in 
philosophy related to the federal role in America altered many 
things and one of them was the relationship of the federal 
government and the states. Rather than the federal government 
mandating day care requirements, the focus changed with the 
locus of control moving from the federal level to the state level 
via block grant funding with very few federal requirements. 
This meant a moratorium to FIDCR and its ultimate demise. 
The federal government was not going to be in the business of 
providing day care, this was going to be the jurisdiction of the 
states. Head Start did become the exception to this rule with its 


own standards and monitoring system. 


The focus of federal funding switched from the national to the 


state level in determining compliance with each state’s 


respective child care licensing rules and not with an overarching 
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FIDCR. There was still interest in making these state 
monitoring systems as effective and efficient but there was no 
interest in the federal government determining what these 
requirements would be. Two monitoring approaches grew out 
of this need for effectiveness and efficiency: risk assessment 
and key indicators. | These two approaches were originally 


designed and implemented as part of a federally funded project 


called the Children’s Services Monitoring Transfer Consortium 


in which a group of five states: New York, Michigan, 
Pennsylvania, West Virginia, and California teamed up to 
explore their most effective and efficient monitoring systems 


and begin transferring these systems to one another and beyond. 


These two monitoring approaches were tested in the above 
respective states and it was determined that their impact had a 
positive effect on the children who were in those day care 
centers. This was a major finding, similar to the FIDCR 


appropriateness study, in which these approaches provided 
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safeguards related to the health and safety of children while in 


day care. 


1990s 


By the 1990s, it became clear that the federal government had 
pretty much drawn back from any leadership role in having 
mandated federal requirements when it came to health and 
safety in child care. It was left to national ECE advocates who 
were positioned within the federal government (Administration 
for Children and Families; Maternal and Child Health Bureau) 
as well as throughout the USA with national and state agencies 
and organizations (American Academy of Pediatrics; American 
Public Health Association, National Resource Center for Health 
and Safety in Child Care) that saw a need for child care health 
and safety recommendations at least. If we could not have 


requirements, we could at least have recommendations and 


provide guidance to child care programs throughout the USA. 


This led to the first edition of Caring for Our Children which 


was a comprehensive set of child care health and safety 
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standards. It was a major game changer for the ECE field 
because now there was a universal set of standards based upon 
the latest research literature for states to use as they considered 
revising and updating their respective state licensing child care 


rules. 


But there was a problem. Caring for Our Children was a 
comprehensive set of health and safety standards which was 
their strength but at the same time it was their weakness. They 
were so comprehensive (well over 500 well researched 
standards) that they were intimidating and it was difficult to 


determine where to begin for the states. 


Several researchers remembered the two approaches to 


monitoring designed in the previous decade and wondered if 


they could be helpful in focusing or targeting which of the 


standards were the most critical/salient standards. The risk 
assessment approach to monitoring appeared to have the most 
immediate applicability and Stepping Stones to Caring for Our 


Children was born. This document clearly articulated which of 
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the 500+ Caring for Our Children standards placed children at 
greatest risk for mortality or morbidity by not being in 
compliance with the respective standard. Since the early 1990s, 
Caring for Our Children and Stepping Stones to Caring for Our 
Children have gone through three editions and have become 
very important resources to state licensing agencies as they 


revise, update and improve their ECE rules. 


2000s 


In this decade several federal and national organizations began 
to use Caring for Our Children standards in innovative ways to 


measure how well ECE looked at a national level. The 


Assistant Secretary’s Office for Planning and Evaluation in the 


U.S. Department of Health and Human Services published the 
Thirteen Indicators of Quality Child Care based upon a core set 
of predictor standards from Caring for Our Children. These 
were standards that predicted overall compliance with all the 
standards and were seen as an efficient monitoring system. 


NACCRRA (National Association for Child Care Resource and 
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Referral Agencies) began publishing a national report card on 
how well states met specific standards and monitoring protocols 
based upon similar predictor standards from Caring for Our 


Children. 


These efforts helped states to make significant changes in their 
ECE rules in their respective states and in a very voluntary way 
suggested a means for national standards for the ECE field 
although we would need to wait until the next decade in order 
to see such a published document of national ECE health and 
safety standards for early care and education: Caring for Our 


Children Basics. 


2010s 


By the 2010s, ECE had grown into a very large but unwieldly 
assortment of programs with varying levels of quality. Again 
because of major federal funding, the Child Care Development 


Block Grant, along with changes and enhancements in 


professional development, accreditation systems, quality rating 


and improvement systems, the ECE landscape had become 
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more complex and less easy to navigate. And rather than 


coming together it was clearly more fragmented than ever. 


We had very minimal requirements for the federal funding and 
most of these requirements were geared to the state agency 
using the state’s respective licensing rules as the threshold for 
standards. This approach worked well with states with 
excellent licensing rules, but it wasn’t working as well with 
states who did not have equally excellent licensing rules. We 
still did not have a core set of standards for ECE programs. 
Enter Caring for Our Children Basics which took the best 
aspects of the above two monitoring approaches, risk 


assessment and key indicators and molded it into this new 


document. This work was led by the federal government’s 


Administration for Children and Families, U.S. Department of 
Health and Human Services and although the standards are still 
recommendations and guidance, it is our best attempt at having 
national standards for early care and education. It is an attempt 


to provide guidance to the full ECE field, child care, Head Start, 
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preschool, and center based as well as home-based care. It 
would be nice to have Caring for Our Children Basics as the 


health and safety foundation for early care and education 


throughout the USA. I don’t see this happening in my lifetime. 


2020s: Looking to the Future 


As a footnote to this essay, the new decade has been dealt with 
a major curve ball with COVID19 rearing its ugly head and 
ECE has been impacted greatly because of this pandemic. As 
of this writing we are nowhere closer to a solution to getting 
ECE programs back on line. If anything, the pandemic really 
demonstrated the fragility of the ECE system we have built over 
the past 50 years and it clearly has not done very well. My hope 
is that we can learn from the past 50 years and not continue 
another 50 years along the same route; although I am guessing 
that many ECE advocates would be glad to have what we had 
before the pandemic because what we have right non- 
sustainable. We know a lot more today than what we knew 


back in 1965 when we were worried about would day care hurt 
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children’s development. We know today that quality ECE 


benefits children but unfortunately, we are no closer to attaining 


this today than we were 50 years ago. 


Two programs that have been very successful in avoiding these 
pitfalls are Head Start and the national Military Child Care 
program. Both programs are exemplary examples of quality 
early care and education being provided with separate funding 
streams and _ standards. Interesting enough when the 
Administration for Children and Families published Caring for 
Our Children Basics, both these programs were part of the 
reach of the published standards. As we re-invent and re- 
structure ECE we should be looking to both these very 


successful programs for guidance. 


The above history and timeline have been drawn from the early 
care and education field but very honestly that is where the 
majority of the research and development has been over the past 


50 years. There have been some excursions into other human 
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services but they don’t have the details as with early care and 


education. 
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Chapter 2 


Conceptual and _ Theoretical Underpinnings, 


Program Monitoring Paradigms 


This second chapter provides the learner with the key 
conceptual and theoretical foundations related to licensing 
measurement. As you have seen from the first chapter and will 
continue to see throughout this book, licensing measurement 
does have some idiosyncrasies which are not present in other 


data distributions. 


Well, the same thing can be said when it comes to the 
conceptual and theoretical underpinnings. One of the first 
limitations that will be noted is the regulatory compliance 
theory of diminishing returns which has _ tremendous 
implications when implementing and enforcing rules. It had 


always been assumed that full 100% regulatory compliance 
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with rules was what made a high-quality program. However, in 
the late 1970's and into the early 1980's, it became clear that this 
was not the case. When this hypothesis was tested it became 
clear that moving from low regulatory compliance to 
substantial regulatory compliance did demonstrate that program 
quality differed significantly in the substantial regulatory 
compliant programs being of a higher quality than those of 
lower regulatory compliance. However, when one moved from 
the substantial regulatory compliance level to the full 100% 
regulatory compliance level, there was a definite plateauing 


effect in which the programs were not increasing in quality as 


previously and in some cases, actually decreased in quality. 


This above result was surprising and very controversial when it 
was first published in the mid 1980's. Many, if not most, 
regulatory compliance specialists did not agree with the finding. 
However, this relationship has held up in many other studies 


conducted since then and in other human service areas. It 
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became the new rule in clearly demonstrating if not a decline, 
always a plateauing effect in moving from substantial to full 
compliance. Today because of all these supporting studies, the 
result is generally accepted and has influenced public 
regulatory compliance policy formulation throughout the 


world. 


This regulatory compliance theory of diminishing returns has 
had tremendous impacts in how we have come to measure 
regulatory compliance in the licensing field. Rather than 
viewing it in a linear modality, it suggested that a more targeted, 
non-linear modality or metric might be more effective and 
efficient. Rather than focusing on full regulatory compliance it 
suggested that a key indicator, abbreviated, or targeted 


monitoring of rules was a better approach. 


Without the regulatory compliance theory of diminishing 


returns, the focus on what has become differential monitoring 
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or targeted monitoring would never have occurred. There would 
have been no need to move from always requiring full 100% 
regulatory compliance with all rules. This is a very important 
distinction and you, the learner, will see many applications and 


implications as you move through the chapters in this text. 


Moving from the Theory to the Conceptual 


Conceptually, licensing measurement is built around obviously 
licensing but there are other systems which impact on licensing 
which are demonstrated in the first licensing measurement class 
when one compares the various regulatory and non-regulatory 
systems in the Morgan Model - Methods for Achieving Quality 
Child Care. There are contractual systems, such as QRIS 


(Quality Rating and Improvement System) or other types of 


quality initiatives. These other types of quality initiatives are 


non-contractual systems, such as professional development or 
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training or technical assistance systems; or accreditation 


systems. 


These above systems can be integrated into a unified model 
called the arly Childhood Program Quality 
Improvement/Indicator Model or Differential Monitoring 
Logic Model and Algorithm (ECPQI2M/DMLMA) which is 
depicted in the lecture slides from the NARA Licensing 
Measurement Course that accompanies this text if you desire to 
utilize those resources and is detailed in several of the handouts. 
Since this will become the unifying framework when discussing 
licensing measurement, I would suggest that you as the learner 
spend a good deal of time reviewing those slides and handouts. 
I would think that you will want to return to them as you move 
through the upcoming chapters and classes as part of the NARA 


course to make certain you continue to understand how all the 


disparate pieces fit together into a uniformed whole. 
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By using the ECPQI2M/DMLMA (also see chapter 5 which 
provides a more detailed step by step guide for the development 
within a state licensing agency) it offers all the key elements to 
building an effective and efficient program monitoring system 
by integrating regulatory compliance and program quality and 
professional development systems along with differential 


monitoring's risk assessment and key indicator methodologies. 


There are readings related to professional development that are 
important components to making sure that the ECPQI2M is 
working as it should. One of the consistent key indicators deals 
with professional development/training. There are examples of 
creative and innovative ways the training can be delivered over 
the internet. Pay particular attention to the iLookOut program, 
especially to its delivery system. Check out the 


https://RIKInstitute.com/publications/. website for these 


publications and reports, there are several articles that describe 
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the program as well as its innovative cognitive mapping and 


online delivery platform. 


Program Monitoring Paradigms 


This section provides some key elements to two potential 
regulatory compliance monitoring paradigms 
(Differential/Relative versus Absolute/Full) for regulatory 
science based upon the Regulatory Compliance Theory of 


Diminishing Returns. 


As one will see, there is a need within regulatory science to get 
at the key measurement issues and essence of what is meant by 
regulatory compliance. There are some general principles that 
need to be dealt with such as the differences between individual 
rules and rules in the aggregate. Rules in the aggregate are not 
equal to the sum of all rules because all rules are not created nor 


administered equally. And all rules are to be adhered to, but 


Licensing Measurement & Monitoring Systems Fiene 


there are certain rules that are more important than others and 
need to be adhered to all the time. Less important rules can be 
in substantial compliance most of the time but important rules 


must be in full compliance all of the time. 


Rules are everywhere. They are part of the human services 
landscape, economics, banking, sports, religion, transportation, 
housing, etc... Wherever one looks we are governed by rules in 
one form or another. The key is determining an effective and 
efficient modality for negotiating the path of least resistance in 
complying with a given set of rules2. It is never about more or 
less rules, it is about which rules are really productive and 
which are not. Too many rules stifle creativity, but too few rules 
lead to chaos. Determining the balance of rules is the goal and 


solution of any regulatory science paradigm. 


Differential/Relative versus Absolute/Full Regulatory 


Compliance Paradigms: this is an important key organizational 
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element in how rules are viewed when it comes to compliance. 
For example, in an absolute/full approach to regulatory 
compliance either a rule is in full compliance or not in full 
compliance. There is no middle ground. It is black or white, no 
shades of gray as are the cases in a differential/relative 
paradigm. It is 100% or zero. In defining and viewing these two 
paradigms, this dichotomy is the organizational key element for 
this paper. In a differential/relative regulatory compliance 
paradigm full compliance is not required and emphasis on 


substantial regulatory compliance becomes the norm. 


Based upon this distinction between differential/relative and 


absolute/full regulatory compliance paradigms, what are some 


of the implications in utilizing these two respective approaches. 


Listed below are the basic implications that occur when 
selecting either of the two approaches on program monitoring 
systems: differential/relative versus absolute/full regulatory 


compliance paradigms. 
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There are ten basic implications that will be addressed: 1) 
Substantial versus Monolithic. 2) Differential Monitoring 
versus One size fits all monitoring. 3) “Not all standards are 


created equal” versus “All standards are created equal”. 4) “Do 


things well” versus “Do no harm’. 5) Strength based versus 


Deficit based. 6) Formative versus Summative. 7) Program 
Quality versus Program Compliance. 8) 100-0 scoring versus 
100 or O scoring. 9) QRIS versus Licensing. 10) Non-Linear 


versus Linear. 


1) Substantial versus Monolithic: in monolithic regulatory 
compliance monitoring systems, it is one size fits all, everyone 
gets the same type of review (this is addressed in the next key 
element below) and is more typical of an absolute paradigm 
orientation. In a substantial regulatory compliance monitoring 
system, programs are monitored on the basis of their past 


compliance history and this is more typical of a relative 
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paradigm orientation. Those with high compliance may have 
fewer and more abbreviated visits/reviews while those with low 


compliance have more comprehensive visits/reviews. 


2) Differential Monitoring versus One Size Fits All Monitoring: 
how does this actually look in a program monitoring system. In 
differential monitoring (Differential/Relative Paradigm), more 
targeted or focused visits are utilized spending more time and 
resources with those problem programs and less time and 
resources with those programs that are exceptional. In the One 
Size Fits All Monitoring (Absolute/Full Paradigm), all 
programs get the same type/level of review/visit regardless of 


past performance. 


3) “Not all standards are created equal” versus “All standards 


are created equal”: when looking at standards/rules/regulations 
it is clear that certain ones have more of an impact on outcomes 


than others. For example, not having a form signed versus 
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having proper supervision of clients demonstrates this 
difference. It could be argued that supervision is much more 
important to the health and safety of clients than if a form isn’t 
signed by a loved one. In a differential/relative paradigm, all 
standards are not created nor administered equally; while in an 
absolute/full paradigm of regulatory compliance, the standards 


are considered created equally and administered equally. 


4) “Do things well” versus “Do no harm” (this element is dealt 


with in the 4" chapter below as well): “doing things well” 


(Differential/Relative Paradigm) focuses on quality of services 
rather than “doing no harm” (Absolute/Full Paradigm) which 
focuses on protecting health and safety. Both are important in 
any regulatory compliance monitoring system but a balance 
between the two needs to be found. Erring on one side of the 
equation or the other is not in the best interest of client 
outcomes. "Doing no harm" focus is on the "least common 


denominator" — the design and implementation of a monitoring 
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system from the perspective of focusing on only 5% of the non- 
optimal programs ("doing no harm") rather than the 95% of the 


programs that are "doing things well”. 


5) Strength based versus Deficit based: in a strength-based 


monitoring system, one looks at the glass as “half full” rather 


than as “half empty” (deficit-based monitoring system). 


Emphasis is on what the programs are doing correctly rather 
than their non-compliance with standards. A strength-based 
system is non-punitive and is not interested in catching 
programs not doing well. It is about exemplars, about excellent 
models where everyone is brought up to a new higher level of 


quality care. 


6) Formative versus Summative: differential/relative regulatory 
compliance monitoring systems are formative in nature where 
there is an emphasis on constant quality improvement and 


getting better. In absolute/full regulatory compliance 
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monitoring systems, the emphasis is on being the gate-keeper 
(more about the gate-keeper function in the next section on 
regulatory compliance/licensing and program quality) and 
making sure that decisions can be made to either grant or deny 
a license to operate. It is about keeping non-optimal programs 


from operating. 


7) Program Quality versus Program Compliance: (this element 
is dealt with in greater detail in the fourth chapter) 
differential/relative regulatory compliance monitoring systems 
focus is on program quality and quality improvement while in 
absolute/full regulatory compliance monitoring systems the 


focus in on program compliance with rules/regulations with the 


emphasis on full, 100% compliance. 


8) “100 — 0 scoring” versus “100 or O scoring”: in a 
differential/relative regulatory compliance monitoring system, 


a 100 through zero (0) scoring can be used where there are 
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gradients in the scoring, such as partial compliance scores. In 
an absolute/full regulatory compliance monitoring system, a 
100% or zero (0) scoring is used demonstrating that either the 
standard/rule/regulation is fully complied with or not complied 
with at all (the differences between nominal and ordinal 
measurement is dealt with in the next section on regulatory 


compliance/licensing and program quality). 


9) QRIS versus Licensing: examples of a differential/relative 
regulatory compliance monitoring system would be QRIS — 
Quality Rating and Improvement Systems. Absolute/full 
regulatory compliance systems would be state licensing 


systems. Many programs talk about the punitive aspects of the 


present human services licensing and monitoring system and its 


lack of focus on the program quality aspects in local programs. 
One should not be surprised by this because in any regulatory 
compliance system the focus is on "doing no harm" rather than 


"doing things well". It has been and continues to be the focus of 
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licensing and regulations in the USA. The reason QRIS - 
Quality Rating and Improvement Systems developed in early 
care and education was to focus more on "doing things well" 
rather than "doing no harm". This is not the case in many 
Canadian Provinces and European countries in which they have 
incorporated program quality along with specific regulatory 


requirements. 


10) Non-Linear versus Linear: the assumption in both 
differential/relative and absolute/full regulatory compliance 
monitoring systems is that the data are linear in nature which 
means that as compliance with rules increases, positive 
outcomes for clients increases as well. The problem is the 
empirical data does not support this conclusion. It appears from 
the data that the relationship is more non-linear where there is 
a plateau effect with regulatory compliance in which client 


outcomes increase until substantial compliance is reached but 


doesn’t continue to increase beyond this level. There appears to 
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be a “sweet spot” or balancing of key rules that predict client 
outcomes more effectively than 100% or full compliance with 
all rules — this is the essence of the Theory of Regulatory 
Compliance — substantial compliance with all standards or full 
compliance with a select group of standards that predict overall 


substantial compliance and/or positive client outcomes. 


As the regulatory science and administrative fields in general 
continue to think about the appropriate monitoring systems to 


be designed and implemented, the above structure should help 


in thinking through what these measurement systems’ key 


elements should be. Both paradigms are important, in particular 
contexts, but a proper balance between the two is probably the 
best approach in designing regulatory compliance monitoring 


systems. 
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Chapter 3 


Instrument Design, Reliability and Validity, 


Statistical Methods and Databases 


This third chapter provides the learner with the key principles 
of instrument design as it relates to licensing measurement. As 
you have seen there are idiosyncrasies’ conceptually and 
theoretically and there are limitations as well, when it comes to 
instrument design. A major limitation with licensing data is that 
it is basically, nominal in nature. It fits the format of Yes or No 
responses. It is not ordinal in any fashion, or at least it hasn't 
been for the past 50 years. In fact, it is only in the past 30 or so 
years that licensing data moved from being predominantly 
qualitative to quantitative. This change started in the 1980's 
with the publication of Instrument based program monitoring. 


Prior to that most licensing studies were written as social work 
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case studies with a great deal of narrative detail but short on 


data utilization that could be used at the macro level. 


Instrument based program monitoring has its critics who are not 
overly excited about its checklist type approach. However, if a 
state is going to track where there are specific issues related to 
regulatory compliance it will be difficult unless an 
instrument/tool/checklist is not used in data collection. If there 
is continued reliance on narrative reports solely it will be 


difficult if not impossible to find any real patterns in the data. It 


is possible with the latest developments in qualitative analyses 


but it is not recommended as the sole means for tracking 
regulatory compliance. I prefer a mixed methods approach 
which focuses on the strengths from both the quantitative and 


qualitative and combines both together. 


Without an instrument-based program monitoring approach it 


would be impossible to utilize the risk assessment and 
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especially the licensing key indicator predictor methodologies. 
In fact, it is really a pre-requisite for designing and 
implementing a targeted monitoring or differential monitoring 


approach. 


In instrument design it is important to utilize the triangulation 
measurement strategy that looks for observation first, followed 
by record/document review, and then lastly by doing interviews 
of staff or parents. The majority of data collection should be 
through observations made in the classroom or facility. When 
observations cannot be made look for policies, files, documents 


that contain the necessary data and then lastly do interviews. 


Reliability and Validity 


This section provides the learner with the key principles of 
reliability and validity which are the mainstay of any 


measurement system. Without these two key principles we do 
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not have a measurement system we can rely on. Reliability 
deals with consistency across inspectors to make certain that 
what is to be measured is measured accurately. Validity 
demonstrates that the system is working as it is supposed to. 
The results are what should be expected from a licensing or 


regulatory compliance system. 


The readings and handouts provide many examples of 
validation studies conducted in the past decade demonstrating 
the validity and reliability of the licensing key indicator 
predictor and risk assessment methodologies (State of 
Washington and the Province of Saskatchewan are the best 


examples of these validation studies). 


Since the large influx in the use of these methodologies over the 


past couple of decades it was incumbent upon us to determine 


if these methodologies were both reliable and valid. Based upon 


these validation studies, it can now be said with a great deal of 
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certainty that the methodologies do what they were intended to 
do. They statistically predict overall regulatory compliance and 
they focus on those rules that place children in greatest risk of 
morbidity or mortality keeping them safe. So, the tenet, which 
will be emphasized throughout this course "Do No Harm" is 


upheld! 


The NARA Licensing Measurement course provide the lecture 
slides where an overview and the key elements to doing 
validation studies, while the readings and handouts provide 
more of the details and the results from these studies. 


(https://www.naralicensing.org/key-indicators) 


Statistical Methods and Data Bases 


This section deals with the statistical methods used and the 
construction of the databases in licensing. As I have said 


repeatedly in my writings over the years there are many 
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limitations related to licensing measurement. The statistical 
methods that can be used with licensing data are limited also, 
because we are dealing with nominal data that are severely 
skewed. Non-parametric statistics is warranted and to deal with 
the severely skewed data, dichotomization of the data base is 


warranted. 


Dealing with data that are not normally distributed poses some 
real challenges in analyzing licensing data sets. It is paramount 
that one runs basic descriptive statistics in assessing the mean, 
standard deviation, variance, skewness, and kurtosis. It will 
help in identifying how badly the data has outliers in a very 
quantitative manner. It will also help in determining where the 


cut scores or thresholds should be for defining the high 


regulatory compliance and the low regulatory compliance 


groups. The Fiene Licensing Predictor Rules and _ their 
respective Fiene Coefficients are determined by using the phi 


coefficient in determining correlations between each rule and 
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the high/low groups for regulatory compliance. This is a 
statistic used with nominal data and is used a great deal in the 
tests and measurement research literature invalidating testing 


procedures. 


The databases should be saved in .csv formatting from an Excel 
file. It is easier to import a .csv file into SPSS or PSPP which is 
the preferred statistical package for conducting these analyses. 
But definitely any statistical package can be used as well, such 
as SAS for example. Outside of generating Fiene Coefficients, 
there are no other statistical techniques that are needed in 


analyzing the database. 


The readings list (https://RIKInstitute.com/publications/) 


provides most, if not all, of the technical research notes 


generated by the Research Institute for Key Indicators. These 
tech research notes provide the latest and most up to date 


information about any changes in the methodologies for 
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generating licensing key indicator predictor rules and risk 
assessment rules. These technical research notes are really 
intended for the serious licensing research and regulatory 
scientist to delve into. They provide the specifics to the various 
statistical methodologies with specific algorithms and logic 


modeling. 


But it still important to address some of the specific statistical 
formulae pertinent to licensing and regulatory science data. For 
example, not all statistics will be relevant to licensing data 
because of its measurement limitations. Licensing data are 
nominal in nature with some instances of ordinal measurement. 
And there are other significant considerations, such as the 
skewness of the data distributions in most licensing data, non- 


linear nature of the regulatory compliance data when compared 


to quality data. So let’s start with the most pertinent statistical 


formulae to be addressed when analyzing licensing and 


regulatory compliance data. 
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The first statistic we need to address is that of the skewness of 
the data. The below formula provides the basic formula for 


determining the skewness of the data distribution. 


n 


~ (n=1\(n=2) 


7 


jal 


This formula is called the adjusted Fisher-Pearson standardized 
moment coefficient and is generally used behind the scenes in 


most software. 


Another formula you will encounter with licensing data will be 
for determining the kurtosis. The following formula provides 
the basic formula for determining the kurtosis of the licensing 


data distributions. 


Kurtosis = n * ni (Yi- Y) 4/(Zni (Yi- Y) 2) 2 
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Another statistic that will be of importance is the variance of 
the data distribution. I have included both the population and 
sample variance formulae because in some cases we need to 
draw a sample and in other cases we have the population data. 
Variance is important with licensing data because it is very 
lacking when you really examine the data distributions which is 


not a good thing from a statistical point of view. 


Population Variance | Sample Variance 


N n 
Ya) 
? at 
oe See i=l 
N 
o° = population variance s’ = sample variance 


x, = value of i” element x, = value of i” element 


x 
x 


= population mean = sample mean 


= population size n= sample size 
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One of the most encountered statistic is the ‘““mean” which is the 


average of the data distribution. Unfortunately with most 


licensing data the mean is not as meaningful as the “median” 


which is the mid point of the data distribution. I have included 
both because the mean is so predominant and the median and 
quartiles are more prevalent in licensing research. The reason 
for using the median over the mean is that the licensing data 


distributions are so severely skewed. 


But the mean formula is provided below: 


Mean of Grouped Data: 


Efe 


n 
where: = mean 
~ = frequency of each class 

= mid-interval value of each class 

= total frequency 
sum of the producst of 
mid — interval values and 
their corresponding frequency 


x= 
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In the majority of cases, the following formula for the median 
will be used as a better measure of central tendency and the 
average score for licensing data. Also, the data are nominal in 
measurement which means we will be collecting frequency 
data, the data are not continuous, they are very discrete. Either 
a rule or regulation is in compliance or out of compliance. 
There are no metrics in between these two extremes. Data will 


be organized and displayed in frequency tables or cross-tabs. 


Formula for calculating the median for licensing data: 


The Median for a grouped data 


he formula for finding a median of a given data is 


class size [assurnir ss size to be equal 


m4maths.com * @ @ @ @m4maths 
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Also, keep in mind that the types of analyses you will be able 
to accomplish will be limited because of the nature of the data 
measurement. When it comes to looking at relationships 
between data sets you will be limited to cross-tabulations and 
the use of the chi-square statistic. I have provided the chi- 
square statistic below to be used with licensing data. Other 
statistics which require a normal distribution or a continuous 


distribution cannot and should not be used. 


A methodology that has been successful with licensing data has 


been the dichotomization of the data distribution because of the 


nominal measurement of compliance vs non-compliance. 


Generally dichotomization of data is not recommended nor 
watranted but I have found that this approach is very successful 


with licensing data distributions. 


The formula for Chi-Square: 
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The Formula for Chi Square Is 


oom > (O; — E;)* 


E; 
where: 
c = degrees of freedom 
O = observed value(s) 


E = expected value(s) 


As one can see these statistics provide a basic grounding from 
an analytical point of view but it is limited because of the real 


limitations in the licensing data measurement characteristics. 


The formula for Key Indicators is in the last section of this book 


within the Graphs, Charts, Figures, and Display Section. 
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Chapter 4 


Regulatory Compliance and Program Quality 


This fourth chapter provides the learner with the similarities and 
differences between regulatory compliance and program 
quality. In the second chapter the regulatory compliance theory 
of diminishing returns was presented which demonstrated a 
non-linear relationship between regulatory compliance and 
program quality. In this chapter, additional concepts will be 
presented to deal with this dynamic tension between regulatory 
compliance and program quality and how we can build one 


upon the other. 


In fact, the future of licensing and regulatory compliance will 
be heavily influenced by this relationship between regulatory 
compliance and program quality. Many jurisdictions are 
attempting to build in quality to their rules/regulations. They are 


73 


Licensing Measurement & Monitoring Systems Fiene 


being very creative in either building separate systems (Quality 
Rating and Improvement Systems: QRIS) or attempting to build 
them right into the rules themselves in more of an ordinal 


format. 


QRIS: Quality Rating and Improvement Systems and other 
Quality Initiatives 


This section provides the learner with key examples from the 
program quality arena, such as QRIS and _ professional 
development. The ECPQI2M model presented here has these 
two systems prominently displayed along with the regulatory 
compliance or licensing system. Together they form the solid 
foundation for providing a very effective delivery system of 


services. When these are combined with risk assessment and the 


key indicator methodologies one can add efficiency to the 


effectiveness side of the equation. The next chapter, Chapter 5, 


will get into more details about how to design an ECPQI2M 
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model along with its associated logic model (DMLMA: 


Differential Monitoring Logic Model and Algorithm). 


As was mentioned in the previous section, there is a delicate 
balance between regulatory compliance and program quality. 
At all times, the ECPQI2M is to keep both regulatory 
compliance and program quality in balance, to keep health & 
safety and quality on an even keel; but as we have seen and will 
see later in this course, this balancing act can get out of kilter at 


times. 


One of the publications produced for OPRE about QRIS 
Validation is directly applicable to licensing measurement and 
has been used within this context in the validation studies that 


will be described in this course. This is an important application 


of this new framework when it comes to validation. It is not just 


for QRIS but can be applied to licensing as well. The state of 


Washington has probably some of the best examples. Please 
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check out these resources and readings later on the RIKI 
website for additional examples. Look specifically for the 
Validation Framework Research Brief (Zellman & Fiene, 2012) 
published by OPRE and the state of Washington Research 


Agenda Report (Stevens & Fiene, 2015). 


Regulatory Compliance/Licensing and Quality 


This section of the chapter will delineate the differences 
between regulatory compliance and quality. It will provide the 
essential principles and elements that clearly demonstrate the 
differences and their potential impact on program monitoring. 
Obviously, there is some overlap between this section and the 
above section dealing with regulatory compliance monitoring 
paradigms. When we think about regulatory compliance 
measurement, we are discussing licensing systems. When we 


think about quality, we are discussing Quality Rating and 


Improvement Systems (QRIS), accreditation, professional 
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development, or one of the myriad quality assessment tools, 
such as the Classroom Assessment Scoring System (CLASS) or 
Environment Rating Scales (ERS’s). All these systems have 
been designed to help improve the health and safety of 
programs (licensing) to building more environmental quality 
(ERS), positive interactions amongst teachers and children 
(CLASS), enhancing quality standards (QRIS, accreditation), 


or enhancing teacher skills (professional development). 


There are ten basic principles or elements to be presented (they 


are presented in a binary fashion demonstrating differences): 1) 


“Do no harm” versus “Do good”. 2) Closed system versus Open 
system. 3) Rules versus Indicators. 4) Nominal versus Ordinal 
measurement. 5) Full versus Partial compliance. 6) Ceiling 
effect versus No Ceiling effect. 7) Gatekeeper versus Enabler. 
8) Risk versus Performance. 9) Structural versus Process 


Quality. 10) Hard versus Soft Data. 
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1) Let’s start with the first principal element building off what 


was discussed in Chapter 2, “Do No Harm” versus “Do Good”. 


In licensing, the philosophy is to do no harm, its emphasis is on 
prevention, to reduce risk to children in a particular setting. 
There is a good deal of emphasis on health and safety and not 
so much on developmentally appropriate programming. In the 
quality systems, such as QRIS, accreditation, professional 
development, Environmental Rating Scales, CLASS, the 
philosophy is to do good, its emphasis is looking at all the 
positive aspects of a setting. There is a good deal of emphasis 
on improving the programming that the children are exposed to 
or increasing the skill set of teachers, or improving the overall 


environment or interaction that children are exposed to. 


2) Closed system versus Open system. Licensing is basically a 
closed system. It has an upper limit with full compliance 
(100%) with all rules. The goal is to have all programs fully 


comply with all rules. However, the value of this assumption 
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has been challenged over the years with the introduction of the 
Regulatory Compliance Theory of Diminishing Returns. With 
quality systems, they have a tendency to be more open and far 
reaching where attaining a perfect score is very difficult to come 
by. The majority of programs are more normally distributed 
where with licensing rules the majority of programs are skewed 
positively in either substantial or full compliance. It is far more 
difficult to distinguish between the really best programs and the 
mediocre programs within licensing but more successful in 


quality systems. 


3) Rules versus Indicators/Best Practices. Licensing systems 
are based around specific standards/rules/regulations that either 
are in compliance or out of compliance. It is either a program is 
in compliance or out of compliance with the specific rule. With 


quality systems, there is more emphasis on indicators or best 


practices that are measured a bit more broadly and deal 
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more with process than structure which is the case with 
licensing. It is the difference between hard and soft data as 
many legal counsels term it. There is greater flexibility in 
quality systems. With this said, if we can look at other service 
types, such as adult-residential services, there has been some 
limited success with blending structural and process elements 


but it still remains a measurement issue on the process side. 


4) Nominal versus Ordinal measurement. Licensing systems are 
nominally based measurement systems. Either you are in 
compliance or out of compliance. Nothing in-between. It is 
either a yes or no response for each rule. No maybe or partial 
compliance. With quality systems, they are generally measured 
on an ordinal level or a Likert scale. They may run from | to 3, 


or | to 5, or 1 to 7. There is more chances for variability in the 


data than in licensing which has 1 or 0 response. This increases 


the robustness of the data distribution with ordinal 


measurement. 
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5) Full or None versus Gradients or Gray Area. Building off of 
the fourth element, licensing scoring is either full or not. As 
suggested in the above elements, there is no in-between 
category, no gradient or gray area. This is definitely not the case 
with quality systems in which there are gradients and 
substantial gray areas. Each best practice can be measured on a 
Likert scale with subtle gradients in improving the overall 


practice. 


6) Ceiling effect versus No Ceiling. With licensing there is 
definitely a ceiling effect because of the emphasis on full 100% 


compliance with all rules. That is the goal of a licensing 


program, to have full compliance. With quality systems, it is 


more open-ended in which a ceiling effect is not present. 


Programs have many ways to attain excellence. 
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7) Gatekeeper versus Enabler: Licensing has always been called 
a gatekeeper system. It is the entry way to providing care, to 
providing services. It is a mandatory system in which all 
programs need to be licensed to operate. In Quality systems, 
these are voluntary systems. A program chooses to participate, 
there is no mandate to participate. It is more enabling for 
programs building upon successes. There are enhancements in 


many Cases. 


8) Risk versus Performance: Licensing systems are based upon 


mitigating or reducing risks to children when in out of home 


care. Quality systems are based upon performance and 
excellence where this is rewarded in their particular scoring by 
the addition of a new Star level or a Digital Badge or an 


Accreditation Certificate. 


9) Structural Quality versus Process Quality: when we think 


of structural quality, we generally think of things we can count 
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easily, such as the number of children or teachers present in a 
classroom or the number of smoke alarms, etc. These are items 
that form the basis of rules within a licensing system. However, 
when we think of process quality, we generally think of things 
that are not as ease to measure, such as interactions between 
teachers and children that are warm and engaging. This is much 
more difficult to measure and generally not part of licensing 
systems but rather program quality tools, such as the ERS and 


CLASS tools. 


10) Hard Data versus Soft Data: this dichotomy is similar to 
number 9 structure quality versus process quality but adds a 
small dimension not present in number 9. It deals with the ease 
with which legal counsel can defend a specific rule or standard 
in a court of law. Hard rules or standards are easy to measure 


while soft rules or standards are more difficult to measure or 


evaluate. Again, they fall along the continue of being structural 


versus process oriented as mentioned in 9 above. 
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There has been a great deal of discussion in the early care and 
education field about the relationship between licensing, 
accreditation, QRIS, professional development, and technical 
assistance. It is important as we continue this discussion to pay 
attention to the key elements and principles in how licensing 
and these quality systems are the same and different in their 
emphases and goals, and about the implications of particular 
program monitoring paradigms and measurement strategies. 
For other regulatory systems outside the human services field, 
the same type of model can be applied positioning compliance 
and quality as a continuum one building from the other because 
I feel that with the introduction of more quality into a regulatory 


context will help to ameliorate the ceiling and plateau effect of 


diminishing returns on performance and outcomes. 
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Chapter 5 


Coordinated Program Monitoring, Differential 


Monitoring, Key Indicator, Risk Assessment, and 


Integrative Monitoring 


This chapter demonstrates the national/federal initiatives 
addressing coordinated program monitoring. There are several 
excellent reports produced by the Administration for Children 
and Families (ACF), Office of Child Care (OCC), Office of 
Planning, Research and Evaluation (OPRE), Assistant 
Secretary’s Office of Planning and Evaluation, and the Federal 
Department of Health and Human Services (HHS) which goes 
a long way in addressing this key issue. In any system where 
there are limited resources, we need to be as cost effective and 
efficient as possible. The handouts which accompany this text 
through the NARA Licensing Measurement course will provide 
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you with many examples of how best to do this. These 
handouts/reports are all available of the RIKI and NARA 
websites as well as within the NARA Licensing Curriculum 


(2000). 


With a closed system and limited resources, a coordinated 
program monitoring system is critical to make certain that we 
have the necessary resources to effectively and efficiently 
protect the clients in the facilities we are mandated to license. 
The key term is "Do No Harm". The federal agency reports in 


this class will provide you with the parameters for building a 


program monitoring system that accomplishes this goal. 


The lecture from the NARA licensing measurement course for 
this section of the chapter consists of a slide that builds upon 
Caring for Our Children Basics (CFOCB)(go to 


https://RIKInstitute.com) and how that publication came into 


existence. Personally, I think it is one of the most significant 
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publications (CFOCB) related to early care and education 


(ECE) standards development that has ever been produced. 


CFOCB provides voluntary standards for all ECE to follow. It 
is the very essence of what coordinated program monitoring 1s 
all about in providing basic safeguards for all children while in 


out of home care. 


Differential Monitoring, Risk Assessment, and Key 


Indicators 


This ehandbook text has gotten into the details of differential 
monitoring, risk assessment, and the key indicator 
methodologies. We have tangentially addressed these 
methodologies throughout the text, but this chapter provides the 
step-by-step process of their development and implementation 


(see the following paragraphs). Also, there are several other 
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publications that deal with this detail on the RIKI and NARA 


Websites. 


The first step in utilizing the DMLMA (Differential Monitoring 
Logic Model and Algorithm) for a state is to take a close look 
at its Comprehensive Licensing Tool (CI) that it uses to collect 
violation data on all rules with all facilities in its respective 
state. If the state does not utilize a tool or checklist or does not 
review all violation data than it needs to consider these changes 
because the DMLMA is based upon an Instrument Based 
Program Monitoring System (IPM) which _ utilizes 


tools/checklists to collect data on all rules. 


The second step for the state is to compare their state’s rules 
with the National Health and Safety Performance Standards 


(Caring for Our Children)(available through 


https://RIKInstitute.ccom) to determine the overlap and 
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coverage between the two. This is the first approach to 


validation which involves Standards review. 


The third step for the state if it utilizes a Risk Assessment (RA) 
tool is to assess the relationship between this tool and Stepping 
Stones to determine the overlap and coverage between the two. 
This is a continuation of the first approach to validation which 


involves Standards review. 


The fourth step for the state is to compare the results from the 
CI with the RA tools. This step is the second approach to 
validation which involves Measures. The correlation between 


Cl and RA should be at the .50 level or higher (.50+). 


In the fifth step, if a state is fortunate enough to have a QRIS — 
Quality Rating and Improvement System in place and has 


sufficient program quality (PQ) data available then they will 


have the ability to compare results from their CI tool with their 
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PQ tool and validate outputs by determining the relationship 
between compliance with health and safety rules (CI) and 


program quality (PQ) measures, such as the ERS’s, CLASS, 


CDPES, etc... This is a very important step because very few 


empirical demonstrations appear in the research literature 
regarding this relationship. This step is the third approach to 
validation which involves Outputs. It would be expected that 
lower correlations (.30+) would be found between CI and PQ 
because these tools are measuring different aspects of quality 
such as health & safety versus caregiver-child interactions or 


overall classroom quality. 


The sixth step is for the state to generate a Key Indicator (KI) 
tool from the CI data base. Please see Fiene & Nixon (1985) 
and Fiene & Kroh (2000) publications available through the 
National Association for Regulatory Administration & 
Research Institute for Key Indicators Websites 


(https://RIKInstitute/com) for a detailed explanation of the 
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methodology for generating a KI tool. This step is also part of 
the second approach to validation which involves Measures. 
The correlation between the CI and KI should be very high 
(.70+) because the KI is a subset of predictor rules taken from 
the CI data base. If a state did not want to use the KI 
methodology, a direct comparison could be drawn from The 
Thirteen Indicators of Quality Child Care — this publication is 


available on the RIKI website. 


The seventh step for the state is to use the RA and KI tools 
together to determine overall compliance of facilities and how 
often and which rules will be monitored for future visits. This 
is the basic component of a Differential Monitoring (DM) 
approach and continues the second approach to validation 


(Measures). Also, this step should drive decisions within the 


technical assistance/training/professional development (PD) 


system in what resources are allocated to a particular facility. It 
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would be expected that moderate correlations (.50+) would be 


found amongst RA, KI, DM, and PD. 


The eighth and final step for the state is to compare the results 
from the various monitoring tools (CI, PQ, RA, KI) with any 
child development outcome (CO) data they collect. This is a 
relatively new area and few, if any, states at this point have this 
capability on a large scale. However, as Early Learning 
Networks and Standards are developed, this will become more 
common place. This step is the fourth approach to validation 
which involves Outcomes. The correlations between CI, PQ, 


RA, KI and CO will be on the lower end (.30+) because there 


are so many other variables that impact children’s development 


other than child care facilities. 


Key Element Definitions: CI = state or federal standards, 
usually rules or regulations that measure health and safety - 


Caring for Our Children or Head Start Performance Standards 
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will be applicable here. PQ = Quality Rating and Improvement 
Systems (QRIS) standards at the state level; ERS (ECERS, 
ITERS, FDCRS), CLASS, or CDPES. RA = risk assessment 
tools/systems in which only the most critical rules/standards are 
measured. Stepping Stones is an example of this approach. KI 
= key indicators in which only predictor rules/standards are 
measured. The Thirteen Indicators of Quality Child Care is an 
example of this approach. DM = differential monitoring 
decision making in which it is determined if a program is in 


compliance or not and the number of visits/the number of 


rules/standards are ascertained from a scoring protocol. PD = 


technical assistance/training and/or professional development 
system which provides targeted assistance to the program based 
upon the DM results. CO = child outcomes which assesses how 
well the children are developing which is the ultimate goal of 


the system. 
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Validation is a continuous approach and is not a once and done 
process. States should look at their monitoring systems on an 
on-going basis and make the necessary adjustments as data are 
collected and compared in order to keep program monitoring as 


cost effective and efficient. 


In the readings/handouts, the learner will find several report 
examples which provide the details of the various 
methodologies. There are more than enough examples, so pick 
the ones you are most interested in seeing. For those of you who 
would like to see more, please go to the RIKI website and look 


under the report's webpage for additional examples. 


Just as differential monitoring helped to change the landscape 


of program monitoring in making it more sensitive to targeted 


reviewing, integrative monitoring introduces a new conceptual 


lense in how program monitoring should be done. Just as 
coordinated monitoring focuses more on the type of care 


provided, integrative monitoring focuses on the actual 
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standards and how they should be formatted. It is more of a 
delicate balancing act between regulatory compliance and 
quality programming when it comes to integrative monitoring 
which is very different from coordinated monitoring which 


emphasizes the facility type. 


An interesting future research area is combining differential and 
integrative monitoring into a new approach to program 
monitoring which would emphasize risk assessment, key 
indicators, and quality programming into this new paradigm. 
Once this is done, it would be relatively easy to take those 


results and apply them within a coordinated monitoring 


approach. To a certain extent, Caring for Our Children Basics 


accomplishes this but without the increased emphasis on the 


quality programming side. 
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Integrative Monitoring 


In the previous chapter, Chapter 4, the delineations between 
regulatory compliance and program quality were dichotomized 
showing how they were different and similar. This section of 
Chapter 5 introduces the notion of integrative monitoring where 
regulatory compliance and program quality are joined together 


in a cohesive management system. 


The concept of integrative monitoring was introduced by Freer 
and Fiene (2023) in a Journal of Regulatory Science article in 
which they propose the concept along with specific constraints 


in implementing the approach. 


In the past regulatory compliance and quality programming had 
their respective silos and were not integrated within the 
monitoring function. In fact, in ECE separate quality 


initiatives, such as Quality Rating and Improvement Systems 
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(QRIS), accreditation, and professional development systems 
have been developed and implemented separate from licensing 
systems in most jurisdictions. In some cases, there were 
attempts to integrate the two arenas but generally these meant 
that there was acknowledgement that both existed but standards 
remained separated and assessors measuring compliance with 


the standards or rules were in different departments. 


The Freer and Fiene (2023) model provides a unique means for 
combining regulatory compliance and quality programming 


into one comprehensive, effective, and efficient approach. 


Freer & Fiene (2023). Regulatory compliance and quality 


programming: Constraints and opportunities for integration, 


Journal of Regulatory Science, Volume 11, Number 1, pps 1- 


10. 
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Chapter 6 


What Research Tells Us, What We Don’t Know, 


and Examples 


This ehandbook text has summarized what we know from the 
research literature about licensing measurement. There have 
been several advances in licensing measurement over the past 
couple of decades. Clearly the Regulatory Compliance Theory 
of Diminishing Returns has taken hold of policy development 
in licensing and regulatory administration. We have seen 
statutes change from requiring full 100% compliance in order 
to receive a license to operate to statutes that are requiring 
substantial regulatory compliance with all rules rather than full 
100% compliance. Getting to those right rules rather than more 
or less rules. Caring for Our Children: Basics is an excellent 


example of this approach. 
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Another example is from developing countries, especially in 
Africa as so evidence by the work of researchers in Kenya who 
have utilized the theory of regulatory compliance repeatedly in 
various industries in order to develop and promulgate effective 
and efficient regulatory policy. In fact, we have a lot to learn 
from these initiatives because they are becoming part of a new 


way of doing policy research that is on a cutting edge. 


Licensing key indicators and risk assessment rules are being 
used on a much larger scale as the _ differential 
monitoring/targeted monitoring approach has expanded. The 
latest Licensing Study conducted by NARA and the National 


Center for Early Childhood Program Quality has demonstrated 


that the majority of states are using one of these approaches. 


The differential monitoring approach and its respective 
methodologies have gone through many enhancements in 


dealing with measurement and statistical nuances related to 
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licensing data distributions, such as severe skewness, kurtosis, 
dichotomization of data groups, eliminating false negatives, 
limitations of nominal data analysis, moving from a nominal 
measurement scale to an ordinal measurement scale, identifying 
generic licensing key indicators, and the relationship between 
regulatory compliance & program quality (Chapters 3 & 4 


highlighted this). 


Eliminating false negatives had been a design issue in the key 
indicator methodology when substantial compliance is utilized 
for the high group dichotomization. It is not an issue when 
100% full regulatory compliance is used but there will be 


instances when substantial compliance will have to be used. 


When this occurs a revision to the original methodology and 


algorithms has to be instituted which has been outlined in a 
RIKINotes Post (January 29, 2023). Utilizing this revision 


eliminates or at least mitigates the false negative effect. 
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All these above enhancements are basically dealt with and 
addressed in the RIKI Technical Research Notes found in the 
ECPQIM/DMLMA text as well as on the RIKI website 


Blog/Notes Page (https://RIKInstitute.com/blog/), or the 


National Association for Regulatory Administration website 
which was cited at the beginning of the eHandBook. The 
interested reader should find all these technical research notes 
in one of those venues. Just look towards the end of the 


webpage to find the research notes. 


What Research Doesn't Tell Us 


So, what are the gaps in the research related to licensing 
measurement that licensing researchers and _ regulatory 
scientists should be paying attention to? This text has provided 
some of the key gaps that have been identified to date. One area 
for further research is the relationship between regulatory 


compliance and outcomes for clients. Are clients healthier and 
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safer in highly compliant programs? Are we seeing fewer 
injuries in those programs of high regulatory compliance? This 
is a critical question that still needs definitive research and 


empirical evidence to confirm. 


There still needs to be additional research that continues to 
validate the rules/standards selected, the measures themselves, 
and the relationship between regulatory compliance and QRIS 
systems. There has been considerable movement in the past 
decade with validation studies being completed in many states 
and provinces and this trend needs to continue. The results to 
date definitely appear to validate all these respective 
components in that they are working as expected, but I would 


feel more confident with additional replication studies being 


completed. 
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International, National, and State Examples 


This section provides us with examples mainly through the 
specific tools that have been designed by different jurisdictions 
for the differential monitoring, key indicator and risk 
assessment methodologies described in this text. The readings 
and handouts provide many such examples which are available 


at https://RIKInstitute.com. You will find examples both from 


the USA as well as Canada. The methodologies have really 
taken off in the last decade as demonstrated by the number of 
contracts NARA has entered into with states and provinces 
throughout the United States (Montana, Michigan, Illinois, 
Indiana, Kansas, Florida, New York, Minnesota, California) 
and Canada (British Columbia, Alberta, Saskatchewan). 
Reports written describing these efforts are available on both 


the RIKI and NARA websites. See the graphic display at the 


end of this ebook in the Display, Graphs, Figures, and Charts 


Section. 
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All of these jurisdictions have demonstrated a certain 
consistency when it comes to licensing key indicator predictor 
rules and risk assessment rules. There are common themes that 


have emerged over the past 4 decades. 


Here are key elements that should be present in a high-quality 
early care and education (ECE) program that any parent should 


be looking for when selecting their child care arrangement: 


Qualified ECE teachers. 

There is a stimulating and dynamic classroom 
environment where children are viewed as competent 
learners. 


A developmentally appropriate curriculum is used 


based upon the assessed individual needs of children. 


Opportunities for families and staff to get to know each 


other. 
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e Families receive information on their children’s 
progress regularly using a formal process. 
Early childhood educators encourage children to 
communicate. 
Early childhood educators encourage children to 
develop reasoning skills. 
Early childhood educators listen attentively when 
children speak. 


Early childhood educators speak warmly to children 


You will witness this consistency in the readings you have 


access to at https://RIKInstitute.com. Please check out the 


website because there are numerous publications and reports 
available to you. All the publications are in the public domain, 


so you are free to download them as you see fit. 


The plan is to continue validating the methodologies to make 


certain that they are keeping children healthy and safe and are 


doing no harm. That is the key element of licensing 
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measurement with a focus on health and safety similar to the 
approach taken by the Nuclear Regulatory Commission (NRC) 
in keeping surrounding communities safe where nuclear power 


plants are located. 


As has been repeatedly demonstrated in this ehandbook text, 
there is a delicate balance between regulatory compliance and 
program quality (remember chapter 4). Some industries are 
more geared towards the health and safety side of the equation 
while others seek a more balanced approach of regulatory 


compliance and program quality. I have attempted to address 


both in this text and hopefully have done an equally balanced 


approach in addressing both sides of the equation. It will be 
interesting to see how things play out as regulatory science 
continues to grow as a science and the impact of licensing 
measurement on the development of this very important 


science. 
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Chapter 7 


Future Directions/Next Steps 


This last chapter deals with where do we go from here. What 
are the next steps for licensing measurement. How do we 
combine the quantitative and the qualitative? How do we have 
a mixed methods approach? How do we combine the best 
aspects of regulatory compliance with program quality 
elements? Are there more effective ways to deal with terribly 
skewed data other than dichotomization? Does it make sense 
to move from a nominal to an ordinal measurement scale with 
regulatory compliance? All these are critical questions for the 
field of regulatory science and its accompanying licensing 
measurement. If we are truly going to build a science, we need 
to spend the requisite time on developing and implementing a 
solid scientific measurement strategy that is both reliable and 


valid. 
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Two of the most critical concepts that will need addressing are 
the ceiling effect/plateauing of quality data and the variance in 
the data distribution. The ceiling effect led to the regulatory 
compliance theory of diminishing returns and has had a major 
impact on the regulatory science field. Without a solution to 
this ceiling effect, it will continue to be difficult to distinguish 
between mediocre quality in programs and high quality in 
programs. At this writing, it appears that this ceiling effect is 
an inherent characteristic of regulatory compliance systems. 
This same ceiling effect does not appear to be present in quality- 


based program monitoring systems nor is the lack of variance 


present in those systems as well. 


Lack of variance in data distributions can and should be 
addressed by building off weighting systems that look at risk or 
performance indicators. This can be a very effective and 
efficient approach to increase the variance in a regulatory 


compliance data distribution. But it does need to be robust 
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enough so that differences can be ascertained at the substantial 
regulatory compliance and the full regulatory compliance 
levels. It has always been easy to distinguish between low 
regulatory compliance and substantial regulatory compliance 
but that is not the case with substantial and full regulatory 
compliance when comparisons are made to program quality 


measures, such as the ERSs and CLASS tools. 


And these two above issues lead us to another key balancing act 
between effectiveness and efficiency. As you have seen 
throughout this ehandbook text there are several concepts that 
need to be balanced with other domains in order to be both 
effective and efficient. A good example is the use of risk 
assessment and key indicators together when designing and 
implementing a differential monitoring system. However, it is 


always possible to put this delicate balance out of sync by 


placing too much emphasis on one or the other domain. 
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For example, a jurisdiction could become so efficient in 
utilizing key indicators that the tool has so few indicators that it 
begins to jeopardize the overall effectiveness of the monitoring 
system. And the other scenario is also a concern in which too 
many indicators are included on the tool in which effectiveness 
might increase but efficiency will decrease substantially. 
Finding that correct balance is an individual study into 
attempting various strategies where clients are not placed at 
additional risk but at the same time we don’t want to jeopardize 


the overall quality of the program. 


So this leads us to a balance between regulatory compliance and 


program quality which is the essence of the last sentence of the 


previous paragraph. It is also the essence of integrative 


monitoring which is attempting to focus both on regulatory 
compliance and quality. On the surface, this sounds really easy 
to develop a balance between regulatory compliance and 


program quality; but in reality, it is difficult to pull off. The 
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reason is that licensing and regulations are just not geared to 
deal with program quality. It is all about health and safety and 
focusing on risk aversion. So in moving forward this is going 


to be a difficult balancing act for most jurisdictions. 


What are some of the other issues that we will need to address 
as we move forward? Validation studies are going to be key in 
moving forward as we determine if the monitoring systems we 
have designed are working as we intended. For the one size fits 
all, this probably will not be a heavy lift; but for differential 
monitoring and integrative monitoring I think these are going 
to be heavy lifts for most jurisdictions. These systems are not 
just descriptive based systems but have inferences built in and 


this is always more difficult to validate. 


And in speaking of validation studies, a key validation study 


will be to validate the use of quality indicators which is the new 


kid on the block. Licensing key indicators have been around 
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for some time, but quality key indicators have not and will need 
a good deal of research to determine what really are predictive 
indicators. I think we do have a good start based upon the 
studies that have been done with QRIS, accreditation, and 
professional development, our major quality initiatives in the 
USA. But additional research is still needed to validate the 
initial results. It is interesting to note that the first pilot testing 
and validation of quality key indicators has occurred in a 
Canadian Province and not in the USA. Our hats are off to the 
Province of Saskatchewan for being the first to pilot test and to 


validate this new approach. 


And then there are the key measurement and statistical methods 


that need further development and refinement as it relates to 
licensing data. If we do move regulatory compliance 
attempting to balance it with program quality, there will most 
likely be experimentation in moving from a _ nominal 


measurement to an ordinal measurement scale. This idea has 
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been suggested in the likes of a Regulatory Compliance Scale. 
But it is still theoretical and has not been attempted yet. But for 


the future, this will become an important area of research. 


On the statistical side, there will be the need to develop 
techniques to deal more effectively with skewed data 
distributions, false negatives, and other licensing data 
idiosyncracies. As I have said many times in this eHandBook 
licensing data are very unique. Part of this uniqueness is the 
fact that the data distributions are anything but normally 
distributed with very little variance. This is a major area of 
concern when it comes to statistical analysis and will need to be 
dealt with heads on in the coming years by regulatory scientists 


and licensing researchers. 


This short ehandbook text is a first step in providing that 


scientific base for building a sound regulatory science, but I am 


hopeful that other licensing researchers and regulatory 
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scientists build upon what has been presented and suggested in 


this eBook. 


For those interested in pursuing any of these topics, please don't 
hesitate to go to the RIKI Institute or the NARA websites for 
additional detailed information. Here are the pertinent websites 


for your ease of access: https://rikinstitute.com 


or https://www.naralicensing.org/key-indicators. 


Research has been going on for approximately 50 years when 


the first kernels of what a regional model for monitoring would 


look like as it related to the human services, in particular early 


care and education. I never thought it would lead to its own 
statistical methodologies and altering how licensing and 
monitoring decision making would occur. And definitely did 
not think that "differential monitoring” would be referenced in 
Federal legislation with the re-authorization of CCDBG. And 


the regulatory science field which spans all industries and 
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domains concerned with the application of rules and regulations 
to our everyday existence has only coalesced over the last 20- 


25 years. 


As I said earlier, the purpose of this ehandbook was as a short 
guide for those in the regulatory science and licensing research 
arenas to get a basic understanding of licensing measurement 
and program monitoring. By starting with it and using it in 
conjunction with all the publications and materials on the RIKI 
and NARA websites as well as the NARA Licensing 
Measurement course, it will provide an introduction to the state 


of the art regarding licensing measurement. 


Let me leave you with a Regulatory Compliance Matrix (see 


table below) which summarizes the key points in this 


ehandbook when it comes to principles of regulatory 


compliance measurement, paradigms, and the quality 


continuum; but also points us in the direction for future research 
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as these are the key elements for licensing measurement and 


monitoring systems. 


The principles detail is provided in the appendix in the last 
document listed: Ten Principles of Regulatory Compliance 
Measurement. The paradigms detail can be found in chapter 2 
of this ehandbook; while the quality continuum is in chapter 4. 
There is a good deal of overlap with the 10 principles, the 10 
elements related to paradigms, and the 10 elements of the 
quality continuum. I would suggest focusing on these common 
elements and principles because they are the most significant 
pieces of the puzzle as it relates to regulatory compliance 


measurement. Taken together, these 30 principles and elements 


provide the basic parameters regulatory scientists, licensing 


researchers, and licensing administrators/policymakers should 
be focusing on when it comes to future research studies related 


to regulatory science. 
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Regulatory Compliance Matrix 


Principles 


Paradigms 


uality Continuum 


Lack of variance 


Substantial vs 
monolithic 


Hard vs soft data 


Ceiling effect 


One size fits all vs 
differential 


Full vs partial 
compliance 


Difficulty between 
full and high 


Rules are equal vs 
not equal 


Rules vs indicators 


Nominal 
measurement 


Do things well vs do 
no harm 


Do no harm vs do 
good 


Moving nominal to 
ordinal 


Strength based vs 
deficit 


Open vs closed 
system 


Dichotomization 


Formative vs 
summative 


Structural vs 
process quality 


Lack of reliability 
and validity 


Program quality vs 
compliance 


Risk vs performance 


Skewed data 


100-0 vs 100 or 0 


Nominal vs ordinal 


Ease between high 
and low 


QRIS vs licensing 


Gatekeeper vs 
enabler 


False negatives 


Linear vs non-linear 


Ceiling effect 
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Appendices 


The Relationship between Early Care & Education 
Quality Initiatives and Regulatory Compliance 

Regulatory Compliance, Licensing, and Monitoring 
Measurement Principles: Rule Compliance Versus 


Rule Performance 


What is the Relationship between Regulatory 
Compliance and Complaints in a Human Services 
Licensing System? 

The Implications in Regulatory Compliance 
Measurement When Moving from Nominal to 
Ordinal Scaling 

So Which Is Better: Differential Monitoring & 
Abbreviated Inspections or Comprehensive 


Inspections? 
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The Dichotomization and Bi-Polarization of the 
Matrix Data Base 

Enhanced Dichotomization Model for Generating 
Licensing Key Indicator 

The Relationship of Licensing, Head Start, Pre-K, 
QRIS, Accreditation, and Professional Development 
and their Potential Impact on Child Outcomes 
Policy Commentary: Regulatory Science 
Measurement Issues of Skewness, Dichotomization 
of Data, and Nominal versus Ordinal Data 


Measurement 


A Potential Reason for Skewed Regulatory 


Compliance Data Distribution 
Data Distribution in Regulatory Science 
Ten Principles of Regulatory Compliance 


Measurement 
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These above appendices provide additional detail to the 
above chapters in delving deeper into some of the key 


points made in Chapters 1-7. 


References follow the appendices and RIKINotes listing 
pertinent publications and posts related to licensing 


measurement and monitoring systems. 


Figure, Charts, Graphs, and Displays supporting the text 


in this eHandBook follow the Reference Section. 


For additional updates before the next edition of this 


ehandbook is published, please go to either the NARA 


website https://www.naralicensing.org/key-indicators 


or the RIKI website https://rikinstitute.com/blog/ 
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The Relationship between Early Care & Education 


Quality Initiatives and Regulatory Compliance 


Over the past couple of decades there has been many early care 
and education initiatives, such as Quality Rating and 
Improvement Systems (QRIS), Professional Development, 
Training, Technical Assistance, Accreditation, and Pre-K 
programs to just name a few. Validation and evaluation studies 
have begun to appear in the research literature, but in these 
studies, there has been few empirical demonstrations of the 
relationship between these various quality initiatives and their 


impact on regulatory compliance or a comparison to their 


respective regulatory compliance. This brief technical research 


note will provide examples of these comparisons taken from the 
Early Childhood Program Quality Improvement and Indicator 
Model (ECPQI2M) Data Base maintained at the Research 


Institute for Key Indicators (RIKIIIc). 
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Ihave written about this back in 2014 (Fiene, 2014) in how the 
various quality initiatives were having a positive impact on the 
early care and education delivery system but at that point 
regulatory compliance data were not available. Today, in 2019, 
with many changes and developments in state data systems, this 
is no longer the case. Now it is possible to explore the 
relationships between data from the various quality initiatives 
and licensing. Several states in multiple service delivery 
systems have provided replicable findings in which I feel 
comfortable reporting out about the relationships across the 


data systems. 


What we now know is that there is a positive and statistically 
significant relationship between regulatory compliance and 
moving up the QRIS Quality Levels. In other words, facilities 
have higher compliance in the higher QRIS Quality Levels and 


lower compliance in the lower QRIS Levels or if they do not 


participate in their state’s respective QRIS (F = 5.047 — 8.694; 


p < .0001). Other quality initiatives, such as being accredited, 
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shows higher compliance with licensing rules than those 
facilities that are not accredited (t = 2.799 - 3.853; p < .005 - 


0001). 


This is a very important result clearly demonstrating the 


positive relationship between regulatory compliance and 


quality initiatives. I have some additional state data sets that I 
will add to the ECPQI2M data base and will continue to analyze 


these relationships. 
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Regulatory Compliance, Licensing, and Monitoring 
Measurement Principles: Rule Compliance Versus Rule 


Performance 


The purpose of this short paper is to delineate the parameters of 


regulatory compliance, licensing and monitoring measurement 


principles (throughout this paper the term “regulatory 


compliance” will be used to encompass these principles). 
Regulatory compliance is very unique when it comes to 
measuring it because it is very different from other 
measurement systems and this impacts how one uses various 
statistical analyses. In this paper, the limitations of the 
measurement system will be highlighted with potential 
solutions that have been devised over the past several decades. 
Hopefully this paper will add to the measurement and statistical 
analysis licensing research literature. It is meant for those 
agency staff who are responsible for designing regulatory 


compliance, licensing and monitoring systems. Its focus is the 
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human services but the basic principles can be applied to any 
standards-based system that is based upon a compliance or 


performance model. 


The organization of this paper is as follows. First, let’s 


introduce what is included when we talk about measurement 


principles for regulatory compliance, licensing and monitoring 


systems. Second, provide examples that should be familiar to 
most individuals who have been involved in the human 
services, in particular the early care and education field. Third, 
what are the limitations of these various systems that have been 
identified in the research literature. Fourth, what are some 
potential solutions to these limitations. And, fifth, what are the 
next steps and where do we go to build reliable and valid 
measurement systems dealing with regulatory compliance, 
licensing, and program monitoring as these relate to the human 


services delivery system. 


So, what is included in this approach. I can be any rule, 


regulation, or standard based measurement system. Generally, 
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these systems are focused on a nominally based system, 
sometimes they will be ordinally based. By a nominally based 
system, either the facility being assessed is in compliance with 
a particular set of rules, regulations, or standards or it is not. In 
an ordinally based system, a facility may attain a score on a 
Likert scale, such as 1 through 5 where | is non-optimal and 5 
is excellent. These types of measurement scales involve a 
performance component and are not limited to more of a 
compliance focus as is the case with a nominally based system. 
These distinctions are important as one will see later in this 
paper when it comes to the selection of the appropriate statistics 
to measure data distributions and the subsequent analyses that 


can be undertaken. 


What are examples of these types of systems? For nominally 


based systems, just about all the licensing systems in the USA, 


Canada and beyond employ this type of measurement strategy. 
As has been said in the previous paragraph, either there is 


compliance or there is not. It is very black or white, there are 
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not shades of gray. For ordinally based systems, these systems 
are a bit more diverse. Accreditation, Quality Rating and 
Improvement Systems (QRIS), the new Head Start Grantee 
Performance Management System (GPMS), the Environmental 
Rating Scales, and the CLASS are all examples of ordinally 
based systems based upon a Likert type measurement system. 
There are many others, but as a research psychologist whose 
total career (50 years) has been spent in early care and 


education, this has been the focus of my research. 


The limitations of the above systems are numerous and, in some 
ways, are difficult to find solutions. In the past, these 
measurement systems have focused more on the descriptive 
aspects of data distributions rather than attempting to be 
predictive or inferential. The first major limitation of the data 
from regulatory compliance systems is the fact that the data 


distribution is markedly skewed. What does skew data mean? 


Most data distributions are normally distributed with very few 


occurrences at the extremes with the majority of the cases in the 
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middle section of the measurement scale. IQ is an example of a 
normally distributed data distribution. In a skew data 
distribution, the majority of data are at one end of the data 
distribution, either at the positive end or the negative end of the 
distribution. With regulatory compliance data, it is at the 
positive end with the majority of facilities being in full or 100% 
compliance with the rules. Very few of the facilities are at the 


negative end of the distribution. 


What is the big deal? The big deal is that statistically we are 
limited in what we can do with the data analyses because the 
data are not normally distributed which is an assumption when 
selecting certain statistical tests. Basically, we need to employ 
non-parametric statistical analyses to deal with the data. The 
other real limitation is in the data distribution itself. It is very 
difficult to distinguish between high and mediocre facilities. It 


is very easy to distinguish between high and low performing 


facilities because of the variance between the high performing 


facilities and the low performing facilities. However, that is not 
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the case between high and mediocre preforming facilities. Since 
the majority of facilities are either in full or substantial 
compliance with the rules, they are all co-mingled in a very tight 
band with little data variance. This makes it very difficult to 
distinguish differences in the facilities. And this only occurs 
with regulatory compliance data distributions. As will be 
pointed later in this paper, this is not the case with the second 
measurement system to be addressed dealing with ordinal 


measurement systems. 


There is also a confounding factor in the regulatory compliance 


data distributions which has been termed the theory of 


regulatory compliance or the law of regulatory compliance 


diminishing returns. In this theory/law, when regulatory 
compliance data are compared to program quality data, a non- 
linear relationship occurs where either the facilities scoring at 
the substantial compliance level score better than the fully 
compliant facilities or there is a plateau effect and there is no 


significant difference between the two groups: substantial or 
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fully compliant facilities when they are measured on a program 
quality scale. From a public policy stand point, this result really 
complicates how best to promulgate compliance with rules. 
This result has been found repeatedly in early care and 
education programs as well as in other human service delivery 
systems. It is conjectured that the same result will be found in 


any regulatory compliance system. 


Another limitation of regulatory compliance data is the fact that 
it is measured at a nominal level. There is no interval scale of 
measurement and usually not even an ordinal level of 


measurement. As mentioned above, either a facility is in 


compliance or not. From a statistical analytical view, again this 


limits what can be done with the data. In fact, it is probably one 
of the barriers for researchers who would like to conduct 
analyses on these data but are concerned about the robustness 


of the data and their resulting distributions. 


Let’s turn our attention to potential solutions to the above 


limitations in dealing with regulatory compliance data. One 
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potential solution and this is based upon the theory of regulatory 
compliance in which substantial compliance is the threshold for 
a facility to be issued a license or certificate of compliance. 
When this public policy determination is allowed, it opens up a 
couple of alternate strategies for program monitoring and 
licensing reviews. Because of the theory of regulatory 
compliance/law of regulatory compliance diminishing returns, 
abbreviated or targeted monitoring reviews are possible, 
differential monitoring or inferential monitoring as it has been 
documented in the literature. This research literature on 


differential monitoring has been dominated by two approaches: 


licensing key indicators and weighted risk assessments. 


A second solution to the above limitations deals with how we 
handle the data distribution. Generally, it is not suggested to 
dichotomize data distributions. However, when the data 
distribution is significantly skewed as it is with regulatory 
compliance, it is an appropriate adjustment to the data. By 


essentially having two groups, those facilities that are in full 
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compliance and those facilities that are not in full compliance 
with the rules. In some cases, the fully compliant group can be 
combined with those facilities that are in substantial compliance 
but this should only be employed when there are not sufficient 
fully compliant facilities which is hardly never the case since 
population data and not sampled data are available from most 
jurisdictions. When data samples were drawn and the total 
number of facilities were much smaller, substantial compliant 
facilities were used as part of the grouping strategy. The 
problem in including them was that it increased the false 
negative results. With them not being included, it is possible to 


decrease and eliminate false negatives. An additional 


methodological twist is also to eliminate and not use the 


substantial compliant facilities at all in the subsequent analyses 
which again helps to accentuate the difference scores between 
the two groups of highly compliant and low compliant scoring 


facilities. 
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The next steps for building valid and reliable regulatory 
compliance systems are drawing upon what has been learned 
from more ordinally based measurement systems and applying 
this measurement structure to regulatory compliance systems. 
As such, the move would be away from a strict nominally based 
measurement to more ordinal in which more of a program 
quality element is built into each rule. By utilizing this 
paradigm shift, additional variance should be built into the 
measurement structure. So rather than having a Yes/No result, 
there would be a gradual Likert type (1-5) scale built in to 
measure “rule performance” rather than “rule compliance” 
where a “1” indicates non-compliance or a violation of the 


specific rule. A “5” would indicate excellent performance as it 


relates to the specific rule. A “3” would indicate compliance 


with the specific rule meeting the specifics of the rule but not 


exceeding it in any way. 


This paradigm shift has led to the creation of Quality Rating and 


Improvement Systems (QRIS) throughout the USA because of 
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a frustration to move licensing systems to more quality focused. 
The suggestion being made here is to make this movement 
based upon the very recent developments in designing such 
systems as is the case with Head Start monitoring. Head Start 
GPMS is developing an innovative Likert based ordinal system 
which incorporates compliance and performance into their 
monitoring system. Other jurisdictions can learn from this 
development. It is not being suggested as a replacement for 
QRIS or accreditation or ERS/CLASS assessments but as a 
more seamless transition from licensing to these various 
assessments. As indicated by the theory of regulatory 
compliance and the law of regulatory compliance diminishing 
returns, this relationship between licensing and program quality 


is not linear. By having this monitoring system approach in 


place, it may be able to reintroduce more of a linear relationship 


between licensing and program quality. 
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What is the Relationship between Regulatory Compliance 


and Complaints in a Human Services Licensing System? 


Within licensing measurement and the validation of licensing 
systems it is particularly difficult to have specific outcome 
metrics that can be measured within a human services licensing 
system. The purpose of this technical research note is to propose 


a potential solution to this problem. 


Probably the most accurate measures of licensing outcomes 


focuses on improvements in the health and safety of clients 


within human services licensed facilities, such as: fewer injuries 
(safety) or higher levels of immunizations (health). Another 
measure related to client satisfaction is the number of 
complaints reported about a licensed facility by clients and the 


general public. The advantage of using complaints is that this 
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form of monitoring is generally always part of an overall 
licensing system. In other words, the state/provincial licensing 
agency is already collecting these data. It is just a matter of 
utilizing these data in comparing the number of complaints to 


overall regulatory compliance. 


The author had the opportunity to have access to these data, 
complaint and regulatory compliance data in a mid-Western 
state which will be reported within this technical research note. 
There are few empirical demonstrations of this relationship 
within the licensing research literature. The following results 
are based upon a very large sample of family child care homes 


(N = 2000+) over a full year of licensing reviews. 


The results of comparing the number of complaints and the 
respective regulatory compliance levels proved to show a rather 


significant relationship (r = .47; p < .0001). This result is the 


first step in attempting to understand this relationship as well as 
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developing a methodology and analysis schema _ since 
directionality (e.g., did the complaint occur before or after the 
regulatory compliance data collection?) can play a key role in 
the relationship (this will be developed more fully in a future 
technical research note). The focus of this research note was to 
determine if any relationship existed between regulatory 


compliance and complaint data and if it is worth pursuing. 


It appears that looking more closely at the relationship between 
complaint and regulatory compliance data is warranted. It may 


provide another means of validating the fourth level of 


validation studies as proposed by Zellman and Fiene’s OPRE 


Research Brief (Zellman, G. L. & Fiene, R. (2012). Validation 
of Quality Rating and Improvement Systems for Early Care and 
Education and School-Age Care, Research-to-Policy, 
Research-to-Practice Brief OPRE 2012-29. Washington, DC: 
Office of Planning, Research and Evaluation, Administration 


for Children and Families, U.S. Department of Health and 


Licensing Measurement & Monitoring Systems Fiene 


Human Services) in which four approaches to validation are 
delineated for Quality Rating and Improvement Systems 
(QRIS). This author has taken this framework and applied it to 
licensing systems (Fiene (2014). Validation of Georgia’s Core 
Rule Monitoring System, Georgia Department of Early Care 
and Learning) and more recently proposed as the framework for 


Washington State’s Research Agenda (Stevens & Fiene (2018). 


Validation of the Washington State’s Licensing and Monitoring 


System, Washington Department of Children, Youth, and 


Families). 
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The Implications in Regulatory Compliance Measurement 


When Moving from Nominal to Ordinal Scaling 


The purpose of this paper is to provide an alternate paradigm 
for regulatory compliance measurement in moving from a 
nominal to an ordinal scale measurement strategy. Regulatory 
compliance measurement is dominated by a nominal scale 
measurement system in which rules are either in compliance or 
out of compliance. There are no gradients for measurement 
within the present licensing measurement paradigm. It is very 
absolute. Either a rule is in full compliance to the letter of the 


law or the essence of the regulation or it is not. An alternate 


paradigm borrowing from accreditation and other program 


quality systems is to establish an ordinal scale measurement 
system which takes various gradients of compliance into 


account. With this alternate paradigm, it offers an opportunity 
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to begin to introduce a quality element into the measurement 
schema. It also allows to take into consideration both risk and 
prevalence data which are important in rank ordering specific 


rules. 


So how would this look from a licensing decision making 
vantage point. Presently, in licensing measurement, licensing 
decisions are made at the rule level in which each rule is either 
in or out of compliance in the prevailing paradigm. Licensing 


summaries with corrective actions are generated from the 


regulatory compliance review. It is a nominal measurement 


system being based upon Yes/No responses. The alternate 
measurement paradigm I am suggesting in this paper is one that 
is more ordinal in nature where we expand the Yes/No response 
to include gradients of the particular rule. In the next paragraph, 
I provide an example of a rule that could be measured in moving 


from a nominal to ordinal scale measurement schema. 
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Rather than only measuring a rule in an all or none fashion, this 
alternate paradigm provides a more relative mode of 
measurement at an ordinal level. For example, with a 
professional development or training rule in a particular state 
which requires, let’s say, 6 hours of training for each staff 
person. Rather than having this only be 6 hours in compliance 
and anything less than this is out of compliance, let’s have this 
rule be on a relative gradient in which any amount of hours 
above the 6 hours falls into a program quality level and anything 


less than the 6 hours falls out of compliance but at a more severe 


level depending on how far below the 6 hours and how many 


staff do not meet the requirement (prevalence). Also throw in a 
specific weight which adds in a risk factor and we have a 


paradigm that is more relative rather than absolute in nature. 


From a math modeling perspective, the 1 or 0 format for a Yes 
or No response becomes -2, -1, 0, +1, +2 format. This is more 


similar to what is used in accreditation systems where 0 equals 
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Compliance and -1 and -2 equals various levels of Non- 
Compliance in terms of severity and/or prevalence. The +1 and 
+2 levels equal value added to the Compliance level by 
introducing a Quality Indicator. This new formatting builds 
upon the compliance vs non-compliance dichotomy (C/NC) but 
now adds a quality indicator (QI) element. By adding this 
quality element, we may be able to eliminate or at least lessen 
the non-linear relationship between regulatory compliance with 
rules and program quality scores as measured by the 
Environmental Rating Scales (ERS) and CLASS which is the 
essence of the Theory of Regulatory Compliance (TRC). It 
could potentially make this a more linear relationship by not 


having the data as skewed as it has been in the past. 


By employing this alternate paradigm, it is a first demonstration 


of the use of the Key Indicator Methodology in both licensing 


and quality domains. The Key Indicator Methodology has been 


utilized a great deal in licensing but in few instances in the 
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program quality domain. For example, over the past five years, 
I have worked with approximately 10 states in designing 
Licensing Key Indicators but only one state with Quality Key 
Indicators from their QRIS — Quality Rating and Improvement 
System. This new paradigm would combine the use in both. It 
also takes advantage of the full ECPQI2M — Early Childhood 
Program Quality Improvement and Indicator Model by 
blending regulatory compliance with program quality 


standards. 


A major implication in moving from a nominal to an ordinal 
regulatory compliance measurement system is that it presents 
the possibility of combining licensing and quality rating and 
improvement systems into one system via the Key Indicator 


Methodology. By having licensing indicators and now quality 


indicators that could be both measured by licensing inspectors, 


there would be no need to have two separate systems but rather 


one that applies to everyone and becomes mandated rather than 
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voluntary. It could help to balance both effectiveness and 
efficiency by only including those standards and rules that 
statistically predict regulatory compliance and quality and 


balancing risk assessment by adding high risk rules. 


I will continue to develop this scale measurement paradigm 
shift in future papers but wanted to get this idea out to the 
regulatory administration field for consideration and debate. 
This will be a very controversial proposal since state regulatory 
agencies have spent a great deal of resources on developing free 
standing QRIS which build upon licensing systems. This 


alternate paradigm builds off my Theory of Regulatory 


Compliance’s key element of relative vs absolute measurement 


and linear vs non-linear relationships. Look for additional 
information about this on my website RIKI Institute Blog - 


https://rikinstitute.com/blog/. 
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So Which Is Better: Differential Monitoring & 


Abbreviated Inspections or Comprehensive Inspections? 


During 2019 and 2020, several validation studies have been or 
are being completed in the states of Washington, Indiana, and 
in the Province of Saskatchewan. These validation studies are 
determining if the key indicator and risk assessment 
methodologies are valid approaches to conducting abbreviated 
inspections in comparison to more comprehensive inspections 
in which all rules are assessed. These abbreviated inspections 
are a form of differential or targeted monitoring. This technical 
research note focuses on the empirical evidence to determine 


the efficacy of these approaches, are they better than doing 


comprehensive reviews when it comes to health and safety 


outcomes. 


When the key indicator and risk assessment methods were 
originally proposed in the 1980’s, an outcome validation study 


was completed in Pennsylvania during 1985 — 1987 by Kontos 
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and Fiene to determine what impact those methods had on 


children’s development. In that original study, it was 


determined that the Child Development Program Evaluation 
Indicator Checklist (CDPEIC) was more effective and efficient 
in predicting child development outcomes than the more 
comprehensive Child Development Program Evaluation. In 
fact, the CDPEIC and the accompanying Caregiver Observation 
Scale (COFAS) were as effective and more efficient than the 
ECERS — Early Childhood Environmental Rating Scale in that 


study. 


Fast forward to 2019 — 2020, in the province of Saskatchewan, 
Canada, and a similar study was undertaken but in this case the 
outcomes were more based upon health and safety rather than 
child development developmental outcomes. In this case, again 
the key indicator and risk assessment tool was both a more 
effective and efficient model over the more comprehensive 
inspection approach giving credence to utilizing differential 


monitoring with abbreviated inspections. 
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In both of the above validation studies involving either child 
development assessment outcomes or health & safety 
outcomes, a 16 to 28% increase in effectiveness was observed 
in the outcome data. In the abbreviated or targeted inspections, 
33% of the total rules or less are used to make the determination 
of regulatory compliance. It is like having the best of both 
worlds when it comes to effectiveness (16 — 28% increase in 
outcomes) and in efficiency (66% fewer rules being used). 
These studies help to validate the use of differential monitoring 


as a viable alternative to the more comprehensive one-size-fits- 


all monitoring reviews. 
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The Dichotomization and Bi-Polarization of the Matrix 


Data Base 


This latest technical note updates the thresholds for the high and 
low groups within the key indicator matrix. This technical note 
is based upon the latest studies during the early 2015 time frame 
in which very large data distributions were available to test 
certain criteria with the key indicator methodology. Because of 
the extreme skewness present in licensing/regulatory data, 
certain statistical adjustments need to be made so that the 
analyses performed reflect the distribution of data. One of these 


statistical adjustments is the dichotomization of data which is 


generally not suggested with the exception of very skewed data. 


Since licensing data are so skewed, this adjustment has been 
used throughout the key indicator methodology. However, an 
additional adjustment is now warranted given not only the 
skewness of data but also because of the data being nominal in 


nature. This additional adjustment I am calling the 
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bipolarization of data in order to accentuate the differences 
between the high and low groups within the key indicator 


matrix. 


I have tested several data sets utilizing bi-polarization and 
found that the results are more significant with its use than 
without its use. Please keep in mind that licensing data is very 
different from other forms of data found in the early care and 
education (ECE) research literature. It is not like the ERS or 
CLASS data which is more normally distributed and lends itself 
to more parametric statistical analyses. Licensing data are 
nominal in nature and always very skewed which means that 
more non-parametric methods are warranted, such as phi 
coefficient and dichotomization of data. An example of how 


this actually works may help. 


Licensing data are measured as either being in or out of 


compliance. There is no middle ground, it is not measured on a 


Likert scale. Therefore it is nominal in nature, either it is all 


there or it is not. Licensing data are also measured in the sense 
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that all rules are created equally, in other words, they all have 
the same weight or importance, such as | = compliance; 0 = 
non-compliance. Being in full 100% compliance which means 
0 violations is the goal of a regulatory/licensing system. One 
does not want to see many violations of the rules because this 
will place children at risk of harm and the purpose of an early 
care and education (ECE) licensing/regulatory system is to 
reduce the potential harm to children. In the licensing 
measurement literature, this 100% compliant group is generally 
labeled or considered the high compliant group. With some 
licensing laws which allow substantial but not full 100% 
compliance with the full set of rules, it would then be allowable 


to have possibly 1 or 2 violations and still be considered in this 


high compliant group. The low compliant group has been 


generally any program that had any non-compliance or had 2 or 
more violations. When these two groups were compared to each 
individual rule utilizing the phi coefficient formula it was found 


that a more accurate appr oach was to accentuate or increase the 
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difference between the high and low groups by eliminating the 
intervening violations in following manner: high group of 0 
violations; 1-4 violations being eliminated; 5+ violations 
defined as the low group. This additional bi-polarization of data 
helped to accentuate the differences in calculating the phi 


coefficient and provided a more sensitive key indicator tool. 


Another data distribution issue that should be addressed here 
that justifies the above cutoffs is that there is very little variance 
in licensing/regulatory data. Generally the frequency 


distribution is 20 or less and the average set of rules is over 200 


rules. So the frequency distribution is extremely skewed within 


less than 10% of the potential data distribution. Also, the 
majority of programs are 100% in compliance with all the rules. 
And an additional complication is that the scoring of each rule 
is scored as if it had an equal risk value when in reality the rules 
can place children at either great risk to relatively little risk if 
found non-compliant. These measurement issues are very 


different than in other measurement systems such as ERS or 
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CLASS. The important message to take from this is that rules 
are not a ruler, they do not measure things equally and cannot 
be analyzed or compared to other measurement systems that are 


more normally distributed. 


Although licensing is part of the program quality continuum in 


establishing basic health and safety standards for children, it is 


a system with measurement limitations that can only be 


compared on a nominal basis making several statistical 


adjustments as suggested above necessary. 
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Enhanced Dichotomization Model for Generating 


Licensing Key Indicators 


The licensing key indicator methodology has been evolving 
over the past decade in making it more sensitive to the selection 
process of the specific rules to be included as key indicators. 
Some of the enhancements can occur because of state licensing 
data systems being able to provide population data rather than 
having to select sample data. Because of the nominal nature of 


licensing data and the severe skewness of the data distributions, 


non-parametric statistical approaches need to be employed in 


the analysis of the data. 


A key component in the analysis of the licensing data 
distributions is to dichotomization of the data which is generally 
not warranted but is acceptable with very skewed data 
distributions. The dichotomization that has been most 
successful is a H25/M50/L25 distribution in which H25 


represents the High Group of regulatory compliance, M50 
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which represents the Mediocre or Middle Group of regulatory 
compliance, L25 which represents the Lowest Group of 
regulatory compliance. In the past, the methodology allowed 
for full and substantial compliance within the High Group. This 
decision is no longer recommended. Rather, in order to decrease 
the number of False Negatives, it is now recommended that 
only Full (100%) regulatory compliance is used in defining the 


High Group. This eliminates the possibility of False Negatives. 


By making this above change and in using the full distribution 
of licensing data, it enhances the results for generating the 


licensing key indicator rules. For additional information on this 


modeling please see: Fiene, Richard (2018), “ECPQIM 


National Data Base”, Mendeley Data, V1. 
http://dx.doi.org/10.17632/kzk6xssx4d.1 This data base 
provides the detailed ECPQIM data distributions for the above 
changes. The enhancements increase the phi coefficients and 
reliability in either moving or not moving from abbreviated 


inspections to full comprehensive inspections. This data base 
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also contains clear demonstrations of the efficacy of the 


ECPQIM — Early Childhood Program Quality Improvement 


and Indicator Model as a vehicle for improving early care and 


education programs. 
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The Relationship of Licensing, Head Start, Pre-K, QRIS, 
Accreditation, and Professional Development and their 


Potential Impact on Child Outcomes 


This short paper will provide some thoughts about the various 
public policy initiatives/systems to improve early care and 
education, such as licensing, Head Start, Pre-K, QRIS, 
accreditation, and professional development and their potential 
impact on child outcomes. Early care and education is at a major 
crossroads as a profession in attempting to determine which 
quality initiatives have the greatest impact on children. Results 
are starting to come in from early studies which may provide 
some guidance as policy makers begin making decisions about 


where to focus their limited funding resources. 


Improving early care and education programs has a long public 


policy history as we attempt to find the most cost effective and 


efficient means for attaining this lofty goal. There have been 


many ups and downs over the years where funding was 
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adequate and when it was not, but our desire to accomplish this 
goal has always been front and center. Now, as a profession, we 
are at somewhat of a cross-roads in determining which of the 
many quality initiatives appear to have the greatest impact on 
children’s development. When I refer to children’s 


development, I am looking at the whole child from the 


perspective of a child’s developmental status as well as the 


child’s health and safety. 


Presently we have many quality initiatives to look at which is a 
very good thing since at times in the past we did not always 
have so many choices. Probably the one constant throughout the 
history of early care and education in the past century has been 
licensing or regulations/rule formulation. Some many argue 
that licensing is not a quality initiative but I would suggest that 
licensing has many of the structural aspects of quality that have 
been identified in the research literature. The other quality 
initiatives I will discuss have really started and been 


implemented in the very later part of the 20th century so we are 
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talking about a relatively new science when we think about 
having its intended impact on children. Also, I am talking about 
large public policy initiatives rather than highly structured, 
single focused research studies involving small samples of 


children. 


Let’s start with licensing since this system has been present for 
the longest period of time. The purpose of licensing is to act as 
the gatekeeper to the early care and education field in which 
only those providers who meet specific standards, generally 
called rules or regulations are permitted to operate and care for 
children. The rules are dominated by health and safety concerns 
with less emphasis on curriculum planning and _ staff-child 
interactions. The rules measure more structural aspects of 


quality than the process aspects of quality; dealing with what 


attorney’s call the “hard data” rather than the “soft data”. 


Since licensing rules allow entry into the early care and 
education field to provide services usually the rules are not 


overally stringent with the majority of providers being in high 
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compliance if not full compliance with all the rules. This would 
be expected since these are basic health and safety standards. 
And in fact when one looks at compliance data, it is extremely 
skewed with the majority of providers having very high 
compliance scores with relatively few violations of the rules. 
However, this does introduce a certain difficulty in using these 
data for decision making purposes at an aggregate level because 
so many providers score at a high level it becomes increasingly 
difficult to distinguish between the really excellent providers 
and the somewhat mediocre providers. Another way of looking 
at this skewing of the data is to term it as a plateau effect in 
which there is very little variance at the upper ends of the 
compliance spectrum. This is a major issue with skewed data 
and basic standards which is an important consideration with 


licensing but will also be an important consideration when one 


looks at the other quality initiatives to be addressed shortly. 


Because of this plateau effect with licensing data, it may explain 


much of the lack of relationships found between compliance 
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with rules and any types of outcomes related to children’s 


outcomes and provider’s overall quality. However, with 
licensing data and making comparisons to children’s outcomes 
we should be looking at general health data such as 
immunization status and safety data such as the number of 
injuries at programs with varying levels of compliance with 


health and safety rules. 


A significant development over the past two decades has been 
the development of national health and safety standards with 
the publication of Caring for Our Children (CFOC3) and 
Stepping Stones (SS3). Although these standards are not 
required but are only recommended practice that provides 
guidance to states as they revise their rules, these two 
documents have been embraced by the licensing/regulatory 
administration field. Although unlikely, if not impossible, to 
comply with all the CFOC3 standards, it would be interesting 


to compare states on this set of standards which may add a good 
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deal of variance to the basic health and safety data that has been 


missing with licensing rules. 


The next system to look at is the national Head Start program. 
Out of the major programs that are national in scope, Head Start 
has a long history of providing services to low-income children 
and their families. Head Start Performance Standards are 


definitely more stringent than licensing rules but not as 


stringent as accreditation standards. Based upon Head Start’s 


more stringent standards and the additional supports that are 
part of its program, Head Start generally scores higher on 
program quality tools (e.g., CLASS or ERS) than licensed child 


care in states. 


With Head Start programs, we at times find skewing or 
plateauing of data when we compare compliance with the Head 
Start Performance Standards (HSPS) and program quality tools 
such as the CLASS. However, this is dependent upon the 
various subscales within the CLASS in which the plateauing of 


data does not occur all of the time. I think that has a lot to do 
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with the HSPS being fairly stringent standards as compared to 


state licensing rules in general. 


A program that has gotten a good deal of support at the state 
level are Pre-K programs. These programs come with stricter 
standards than licensed child care with an emphasis on the 
professional development of staff. There is more concern about 
the process aspects of quality which focus more on teacher- 
child interactions. This emphasis on teacher-child interaction 
has paid off in which these programs generally are high 
performers when you compare Pre-K funded classrooms to 
licensed child care classrooms. In fact, Pre-K funding appears 
to have a positive impact on licensed child care in raising 


overall quality scores on the ECERS-R for all classrooms in 


programs that receive Pre-K funding even if some of the 


classrooms are not the direct beneficiaries of the funding. This 
is a very significant finding because we knew that Pre-K 
funding increased the quality of care in classrooms receiving 


those funds, but now, it appears that there is a spillover effect 
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to all classrooms co-located with Pre-K funded classrooms. I 
must admit that I was initially skeptical when Pre-K funding 
was first proposed because I thought it would take funding and 
the focus away from improving licensed child care at the state 
level; but it appears that the advocates for Pre-K were right in 
their assertion that Pre-K would increase the quality of all early 


care and education which includes licensed child care. 


A more recent entry into the state funding scene are QRIS 
(Quality Rating and Improvement Systems) which build upon 


licensing systems, are voluntary, and have substantial financial 


incentives for participating in this quality improvement system. 


It is too early to really determine if QRIS is having the intended 
impact because the program is so new (50% of states have a 
QRIS), and the penetration rate is usually below 50% in any 
given state (remember the system is voluntary). However, in the 
few studies done, the results are mixed. It does appear that 
programs which move up the various star levels do increase the 


quality of care they provide; but in a most recent study looking 
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at child outcomes, no relationship was found between 
increasing levels of compliance with QRIS standards and how 
well children did in those programs with the exception of 
CLASS scores in which teacher-child interactions were 
measured and emphasized — here there were significant 
relationships between higher scores on the CLASS and child 


outcomes. 


Accreditation systems come in many varieties but there are only 
three that I know of in which empirical studies have been done 
to validate their systems: NAEYC, NECPA for centers and 
NAFDC for homes. Also reliability testing has been done in 


each of these systems. Accreditation is a rigorous self-study that 


really improves programs through the self-study process. This 


should come as no surprise because we have known for some 
time that program monitoring all by itself leads to program 
improvements. Now when you couple that with technical 
assistance you see even more improvement. Accreditation is 


usually the other pillar of a QRIS system with licensing being 
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the first pillar. The QRIS standards fill the gap from licensing 
to accreditation. Accreditation is a voluntary system just as in 
most cases with QRIS. However, in accreditation we are 
reaching less than 10% of the programs with the majority of 
these attaining NAEYC accreditation. NECPA and NAFDC 


have much smaller market shares. 


The last system to be addressed is the professional development 
systems that have been established in all states. This is one 
quality improvement initiative that has 100% penetration in all 
states. It is usually tied to QRIS through technical assistance 
and mentoring (coaching). When it focuses on mentoring rather 
than workshops, it has demonstrated its effectiveness in 
changing teachers behaviors in how they interact with children 


in their care in a very positive fashion. This is very important 


because the research literature is clear about the importance of 


the teacher-child interaction when it comes to child outcomes. 
Professional development runs the gamut from pre-service 


(University based programs) to in-service (training, technical 
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assistance, mentoring, coaching) programming for teachers and 


directors. 


So where does this leave us when policy makers begin to try to 
determine which quality improvement initiatives should be 
invested in to start with, which to increase in funding, and 
maybe even which ones should be defunded. I think there are 


some trends we need to begin to look at, such as the following: 


1) Having stringent and rigorous standards is very important. 
The more that we do not, the more opportunities for mediocre 
programs to score artificially higher on whatever scale that is 
used. This is evident with licensing data where the data are 
significantly skewed with a major plateau effect at the upper 


end of compliance rules/regulations. 


2) Emphasis on teacher-child interaction needs to be paramount 
in our quality improvement initiatives. Working with teachers 


through mentoring/coaching appears to be most effective in 


changing teachers’ behaviors in interacting more positively 


with children. 
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3) Making sure we are measuring the right outcomes. Match 
health and safety standards with health and safety outcomes for 
children. Match developmental outcomes for children with 


standards that emphasize positive teacher-child interactions. 


4) Building upon #1 above, find what the key indicators are with 


all the data that we collect. We are spending too much time in 
looking at too many things which in many cases are simply just 
not the right things to look at. As states’ data systems become 
more sophisticated, and they are, this will be easier to do. Let’s 


begin to utilize the data we have already collected. 
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Policy Commentary: Regulatory Science Measurement 
Issues of Skewness, Dichotomization of Data, and Nominal 


versus Ordinal Data Measurement 


The purpose of this policy commentary is to provide some 
context for regulatory scientists in pursuing regulatory policy 
analysis, especially as it relates to regulatory compliance and 
human service licensing data. Regulatory scientists have dealt 
with non-parametric data in the past but in dealing with 
regulatory compliance and human service licensing data are just 
so different from previously measured data in that the nature of 
the data is nominal and extremely skewed to the point that 
several adjustments need to be made in order to analyze the 


data. 


Although the examples being referred to in this policy 
commentary are from the human services field and discipline, I 


am certain that many of the basic concepts presented will 


pertain to other disciplines and fields of study that are impacted 
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by regulatory science. These concepts are not unique to a 
particular discipline but rather are unique to regulatory science 
which has particular parameters, concepts, and truths which are 
pertinent to how regulations/rules/standards are formulated and 


then implemented in various jurisdictions or disciplines. 


There are very logical reasons why regulatory compliance and 
licensing data are so extremely skewed. These data represent 
compliance with basic health and safety rules and regulations 
which provide the basic safeguards for children, youth, and 
adults while being cared for in a form of human services, such 


as child care, youth residential, or adult assisted living care. 


Very honestly a state agency would not want to find their 
regulatory compliance data being normally distributed because 
this would be an indication that the facilities were in low 


compliance with the state's rules and regulations. Having the 


regulatory compliance data be highly negatively skewed is 


actually a good result from a public policy standpoint but not 


from a statistical analytical standpoint. Having 50-60% of your 
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scores within a three-to-five-point range when there may be as 
many as 300-400 data points leaves very little variance in the 
data. It also leads to being very difficult to distinguish between 
the high performers and the mediocre performers. This finding 
has led to a theory of regulatory compliance in which 
substantial compliance but not full compliance with all rules 
and regulations is in the best interests of the clients being served 


(Fiene, 2019). 


In the regulatory science field, this has led to public policies 
emphasizing substantial compliance in order to be a licensed 
human service facility, such as a child care center, youth 
residential program, or an adult assisted living center. The other 
aspect of regulatory compliance and licensing data for 
regulatory scientists to consider is that the data are nominal in 


measurement, either a facility is in compliance or out of 


compliance with a specific rule or regulation. There are no gray 


areas, nO measurement on an ordinal scale. 
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There has been some discussion in the regulatory science field 
for the use of weighted risk assessment methodologies which 
could introduce more variance in the data based upon the 
assumption that all rules or regulations are not created equal nor 
are they administered equally (Stevens & Fiene, 2019). Another 
discussion revolves around the introduction of more program 
quality into the basic health and safety rules and regulations that 
could extend the nominal compliance determination to an 
ordinal scale that goes beyond the basic compliance level 


(Fiene, 2018). 


These measurement idiosyncrasies of regulatory compliance 
and licensing data are presented for regulatory scientists to 


consider if they begin to analyze public policies that involve 


basic health and safety rules and regulations which are very 


different from other public policies being promulgated by state 
and national governments. For the interested reader, an 
international data base for regulatory compliance and human 


services licensing data has been established and maintained by 
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the Research Institute for Key Indicators and Penn State 
University over the past 40 years at the following URL - 


(http://RIKInstitute.com) 


However, the hope is that other disciplines will begin to look at 
their data more closely to determine the natural data 
distributions and ascertain if they are equally as skewed as has 
been found in human service regulatory data. Are you 
measuring the data at a nominal level? Could they be measured 


at an ordinal level based upon a Likert scale? The data being 


referred to are regulatory compliance data which are pegged to 


specific rules/regulations/standards. It is not based upon other 
types of data collected within a regulatory frame of reference, 


such as basic demographic or descriptive data. 
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A Potential Reason for Skewed Regulatory Compliance 


Data Distributions 


One thing that is ever present with regulatory compliance data 
distributions is that they are terribly skewed. See the previous 
post which provides a definition of skewed distributions and 
their implications. This post is going to attempt to provide a 


potential answer to why the data base is skewed. 


At first, I was led to believe that potentially the skewness in the 
data was a result of the rules not being stringent enough, in other 
words, the health and safety standards were too easy to comply 


with. That could definitely be a contributing factor but this is 


not the case in all instances when one compares state human 


service rules and regulations and the Head Start Performance 
Standards. I think a much deeper structure may be operating 


that is more philosophical rather than practical. 
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The philosophy of regulatory compliance and rule formulation 
is one of risk aversion. In other words, how do we mitigate risk 
that potentially increases the chances of mortality or morbidity 
in the clients being served when a specific rule is out of 
compliance. This philosophy emphasizes the elimination of a 
risk, taking something away rather than adding to it. It is 
essentially, "Do No Harm". It is interesting to note that 
generally regulatory compliance scoring is nominal in being 
either "Yes" or "No"; and a lower score is better than a higher 
score, there are fewer violations of rules. Not the way most 


assessment tools are designed. 


For example, when one looks at program quality, this system is 
based upon the open-endedness adding to rather than taking 
away. It is all about, "Do Good" rather than "Do No Harm". 
Generally when you look at the data distributions here, they are 


more normally distributed without the skewed nature of 


regulatory compliance data distributions. Generally program 


quality scoring is ordinal in nature on a Likert Scale. A higher 
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score is better than a lower score. Makes sense in that when you 
have more of a good thing, the higher the score. And the 
philosophy of program quality is one of improvement with 


relatively little emphasis on risk aversion. 


This is an alternate explanation to why regulatory compliance 


data distributions are so terribly skewed in comparison to other 


program quality measures. 
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Data Distributions in Regulatory Science 


Data distributions in the human services as they relate to 
regulatory compliance are generally very skewed distributions 
which means that the majority of facilities being 
assessed/inspected will usually fall very close to the 100% 
compliance level. There will also be an equally large number of 
facilities that are in substantial regulatory compliance (99% - 
98% compliance levels). And then there are much fewer 
facilities that are either at a mid or low level of regulatory 
compliance (97% or lower compliance levels). One might say 
that getting a score of 97% on anything doesn't sound like it is 


mediocre or low but keep in mind we are addressing basic 


health and safety rules and not quality standards. So having 


several health and safety rules out of compliance is a big deal 
when it comes to risk assessment. It could be argued that a state 


licensing agency was not upholding its gatekeeper function by 
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allowing programs to operate with such regulatory non- 


compliance. 


Why is the regulatory compliance data distribution important 
from a statistical point of view. Generally when we are dealing 
with social science data, the data are normally distributed or 
pretty close to being normally distributed. It is a trade mark of 
a well designed assessment tool for example. So when data are 
compared to other normally distributed data, there is a good 
chance that some form of a linear relationship will be 
ascertained, albeit, not reaching statistical significance in many 


cases but linear regardless. 


When a very skewed data distribution is one of the variables as 


in the case with regulatory compliance data and it is compared 
with a normally distributed data set such as a program quality 
tool, ERS or CLASS. Well, the result is generally a non-linear 
relationship with a marked ceiling effect or plateau effect. In 
other words, the data distribution is more curvilinear than 


linear. From a practical standpoint this creates selection 
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problems in the inability to identify the best programs that have 
full regulatory compliance. This can create a public policy 
nightmare in that those programs which are in substantial but 
not full regulatory compliance are as good or in some cases of 
higher quality than those programs in full regulatory 
compliance. The interesting question is does the combination of 
normally distributed data distributions with variables that have 


skewed data distributions always produce this nonlinear result?! 


And lastly, will having two variables that are skewed data 


distributions produce a more random result than if one of the 


two above conditions are present? 
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The Ten Principles of Regulatory Compliance 


Measurement 


The first principle deals with the lack of Variance in data 
distributions. Data are found to be tightly grouped at high 
compliance levels (upper 90% level). This will lead to another 
principle addressed later in this paper dealing with skewness of 
the data distribution. In fact, the majority of scores are at a full 
regulatory compliance level, in other words, 100% in 
compliance with all rules and regulations. This led to variance 


statistics showing little movement and the majority of programs 


being in very close proximity. This makes for difficult 


statistical analyses when there is little variance in the data set. 


The second principle is finding a ceiling or plateau effect in data 
distributions. It was like there was a diminishing returns effect 
as one moves from substantial regulatory compliance (upper 
90%+) to full regulatory compliance (100%) with all rules and 


regulations. This was especially true when one compares the 
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regulatory compliance levels with program quality scores on 
those same programs which is addressed more in the next 


principle. 


The third principle is the difficulty distinguishing levels of 
quality between full and substantial compliance. This principle 
builds off of the previous principle dealing with a ceiling or 
plateau effect. Because so much of the data, as much as 70- 
80% of programs, are grouped so tightly at the substantial and 
high levels of regulatory compliance when one begins to go 
beyond regulatory compliance and begin to look at quality there 
is a great deal of difficulty distinguishing levels of quality. In 
other words, the full regulatory compliant level programs are 


not necessarily the highest quality programs. 


The fourth principle is the fact that rules and regulations are 


measured at a nominal measurement level: the rules and 


regulations are either In-Compliance or Out-of-Compliance. 


The rule or regulation is measured at a “Yes” or “No” level or 


a “1” or “O” level. There are no in-between measures, no 
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ordinal measurement going on. Either you got it, or you don’t. 
It is black or white, no shades of gray. It is just the nature of 
measurement when it comes to rules and regulations which are 
very different in other measurement systems. The data are very 
discrete and not continuous. They are frequency counts and not 
aruler type of measurement. One will not find an interval level 


of measurement in any regulatory science data distribution. 


A fifth principle is attempting to move to an ordinal 
measurement level when quality is included. This principle 
builds off of the previous principle in which in some cases it 
has been suggested to add a quality component to particular 
rules or regulations. This is an interesting development and 
moves the philosophy from one of “Do no harm” to one of “Do 
things well”. It will be interesting to see how much this concept 
moves forward and changes a basic tenet in the regulatory 


science field which is more based upon health & safety, 


gatekeeper, hard data, risk aversion, and deficit based. 
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The sixth principle of regulatory compliance measurement is 
the ability to dichotomize the data can be warranted because of 
the data distribution. Data dichotomization is generally not 
recommended because it accentuates differences in a data set. 
However, given the nature of regulatory compliance 
measurement being at a nominal level, fitting into a bucket 
format, the lack of variance, and the skewness of the data 
distribution all lead to the ability to dichotomization of the data 


set. 


The seventh principle has to do with the problem with false 
negatives and positives, especially false negatives. Because of 
the data being measured in a nominal In-Compliance vs Out-of- 
Compliance dichotomy it can lead to false negatives in which 
In-Compliance decisions are made that in reality are not In- 
Compliance. False positives are a problem as well but not as 
much of a problem as false negatives. In false positives, Out- 


of-Compliance may be determined when in reality the rule or 


regulation is actually In-Compliance. This is not a good 
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scenario for the provider of services, but it potentially doesn’t 


harm the client as much as when a false negative occurs. 


The eighth principle is the lack of reliability and validity testing. 
This principle builds from the previous principle in that there 
are very few examples of scientific testing of instrumentation 
and the administration of protocols to make certain that 
everything is running as it should. Because of this, it leads to 
the above problem of false positives and negatives. All 
jurisdictions need to build in regular reliability and validity 
testing to ascertain that the final decision making is within the 


ranges that are acceptable. 


The ninth principle is the ease in distinguishing levels of quality 
between low and substantial compliance. The one result that 
has been consistent over the years is the ability to see 


differences in programs that score low on_ regulatory 


compliance versus those that are at a substantial or high 


compliant level. From a licensing or regulatory administration 


point of view this is a real plus in being able to be an effective 


183 


Licensing Measurement & Monitoring Systems Fiene 


gatekeeper and keeping non-optimal programs out of service. 
But as indicated in the third principle this advantage is short- 
lived as one moves up the regulatory compliance scale to 
substantial and finally to full regulatory compliance. When one 
gets to these levels it becomes increasingly difficult to 
distinguish differences in quality in those programs that are in 
substantial regulatory compliance versus those that are in full 
regulatory compliance. It appears that the regulatory 
compliance theory of diminishing returns is rearing its 
plateau/ceiling effect. The policy implications are immense 


since the assumption is that there is a linear relationship 


between program quality and regulatory compliance. How do 


we more effectively deal with this non-linear relationship in 


formulating public policy regarding licensing decision making? 


And the final tenth principle is that regulatory compliance data 
are always skewed data. The majority of programs are in 
substantial or full regulatory compliance. And in many cases, 


this can be rather severe. There generally is a long tail which 
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contains some low regulatory compliant programs, but these are 
usually few in number. The data distribution just does not 
approach a normally distributed curve as we see in many other 


examples of social science data distributions. 


It is important as the regulatory science field moves forward 
that we remain cognizant of the limitations of regulatory 
compliance measurement. There are some severe limitations 
that need to be addressed (e.g., skewed data, lack of variance in 
data, ceiling effect, nominal metrics) and building in mitigation 


strategies (e.g., data dichotomization) or it will continue to lead 


to problems in our analyses (e.g., false positives and negatives). 


Licensing Measurement & Monitoring Systems 


The next two papers provide summaries of licensing 


measurement and monitoring systems, and the importance of 


the theory of regulatory compliance; but with the lens of a 


much broader perspective going well beyond the human 


services and including and addressing other industries. 
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Licensing Measurement and Monitoring Systems: 
Regulatory Science Applied to Human Services Regulatory 


Administration Summary 


In the realm of human services regulatory administration, 


ensuring compliance with licensing requirements is crucial for 


maintaining quality standards and safeguarding the well-being 
of individuals receiving care. As regulatory agencies strive to 
enhance their oversight and monitoring capabilities, the 
integration of measurement and monitoring systems has 


emerged as a valuable tool. 


This paper explores the significance of licensing measurement 
and monitoring systems and delves into the application of 
regulatory science in the context of human services regulatory 
administration. It will deal with several issues related to this 
topic and expand its content beyond early care and education 


which has been more of the focus previously. 
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Licensing Measurement & Monitoring Systems 


Licensing measurement and monitoring systems play a crucial 


role in regulatory administration for several reasons: 


Compliance Verification: Regulatory agencies need to ensure 
that businesses and individuals comply with specific laws, 
regulations, and standards. Licensing measurement and 
monitoring systems provide a means to verify compliance by 
collecting data and measuring various parameters. These 
systems help regulators determine whether license holders are 
meeting the required standards and taking appropriate actions 


to mitigate risks. 


Quality Assurance: Licensing measurement and monitoring 


systems contribute to quality assurance efforts by assessing the 


performance of licensed entities. They enable regulators to 


monitor the quality of services and activities associated with the 


licensing process. By establishing measurement criteria and 
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tracking the relevant metrics, regulators can ensure that license 
holders maintain the desired level of quality and meet the 


expectations of consumers or the public. 


Risk Management: Many industries involve inherent risks that 
need to be managed effectively. Licensing measurement and 
monitoring systems allow regulatory agencies to assess and 
monitor the risks associated with licensed activities. By 
continuously monitoring key indicators, regulators can identify 
potential risks, deviations from safety standards, or non- 
compliance issues. This information helps regulators take 


appropriate actions to minimize risks and ensure public safety. 


Data-Driven Decision Making: Licensing measurement and 


monitoring systems generate substantial amounts of data that 


can be analyzed to make informed decisions. Regulators can 


analyze trends, patterns, and performance metrics to identify 


areas of concern or improvement. Data-driven insights enable 
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regulators to make evidence-based decisions, allocate resources 
effectively, and prioritize enforcement actions where they are 


most needed. 


Enforcement and Remediation: When non-compliance or 
deviations from regulations are identified, licensing 
measurement and monitoring systems provide evidence to 
support enforcement actions. Regulators can use the data 
collected to take appropriate enforcement measures, such as 
issuing warnings, imposing penalties, or revoking licenses. 
These systems also help in tracking the progress of remedial 
actions taken by license holders to address any identified issues 


or deficiencies. 


Transparency and Accountability: Licensing measurement and 


monitoring systems enhance transparency and accountability in 


regulatory administration. By implementing these systems, 


regulators can demonstrate their commitment to fair and 
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consistent enforcement of regulations. The data collected and 
analyzed can be made accessible to the public, stakeholders, 
and policymakers, fostering trust, and allowing for external 


scrutiny of regulatory processes. 


Licensing measurement and monitoring systems are vital in 
regulatory administration as they facilitate compliance 
verification, quality assurance, risk management, data-driven 
decision making, enforcement, and accountability. These 
systems help regulators ensure that licensed entities operate 
within the set standards, mitigate risks effectively, and 


safeguard the interests of the public. 


Regulatory Science is relevant to human services regulatory 
administration in all industries. Regulatory science is the 


scientific discipline that combines various fields, including law, 


public policy, data analysis, and risk assessment, to inform and 


guide regulatory decision-making. Measurement and 
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monitoring systems are regulatory science aids in the 
development and implementation of evidence-based 


regulations and policies. 


Regulatory agencies overseeing a wide range of human 
services, such as healthcare facilities, child care centers, mental 
health institutions, and more, face several challenges in their 


oversight role. Some of the key challenges include: 


Diverse and Complex Landscape: The human services sector 


encompasses a broad range of industries, each with its unique 


complexities, regulations, and standards. Regulatory agencies 


must navigate and understand this diverse landscape to 
effectively oversee and enforce compliance. The sheer variety 
of services, settings, and stakeholders involved makes it 
challenging to develop uniform regulations and monitoring 


approaches that address the specific needs of each sector. 
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Rapidly Evolving Practices and Technologies: The human 
services field is constantly evolving, with new practices, 
technologies, and treatments emerging. Regulatory agencies 
need to keep pace with these changes to ensure that the 
regulations remain relevant and up-to-date. However, this can 
be a challenging task, as it requires continuous monitoring, 
research, and adaptation of regulations to address emerging 


risks and advancements adequately. 


Resource Constraints: Regulatory agencies often face resource 
constraints in terms of staffing, funding, and technological 
capabilities. Insufficient resources can limit their capacity to 
conduct thorough inspections, investigations, and monitoring 
activities. Additionally, limited resources may also impact the 


frequency and intensity of oversight, making it difficult to 


identify and address compliance issues effectively. 
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Compliance Variability: Human services facilities and 
institutions can vary significantly in terms of size, ownership, 
resources, and compliance history. Regulatory agencies need to 
develop oversight strategies that account for these variations 
while ensuring consistent enforcement and quality standards 
across the board. Balancing the need for flexibility with the 
need for uniformity is a constant challenge for regulatory 
agencies. And this becomes increasingly complex when 
dealing with the regulatory compliance theory of diminishing 


returns/ceiling effect. 


Stakeholder Engagement and Resistance: Regulatory oversight 


often involves engaging with various stakeholders, including 


facility owners, professionals, service recipients, advocacy 


groups, and the public. These stakeholders may have different 
interests, priorities, and perspectives, leading to potential 


conflicts or resistance to regulatory measures. Balancing the 
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diverse viewpoints and managing stakeholder expectations is 


essential for effective oversight. 


Data Management and Analysis: The vast amount of data 
generated by human services facilities can pose challenges in 
terms of data management, analysis, and interpretation. 
Regulatory agencies need robust systems and processes to 
collect, store, analyze, and make sense of the data to identify 
trends, patterns, and areas of concern. The integration and 
interoperability of data systems across different sectors and 


agencies can be complex and time-consuming. 


Legal and Ethical Considerations: Regulatory agencies must 
operate within legal frameworks and adhere to ethical standards 


while overseeing human services. They need to strike a balance 


between protecting public health and safety and respecting 


individual rights and privacy. Navigating legal complexities, 


ensuring due process, and maintaining confidentiality can be 
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challenging in an environment where ethical dilemmas may 


arise. 


Addressing these challenges requires a proactive and adaptive 
approach from regulatory agencies. They need to foster 
collaboration with stakeholders, invest in capacity-building 
efforts, leverage technology for efficient data management, and 
engage in continuous evaluation and improvement of their 


oversight strategies. 


Inadequate monitoring in the human services can have 


significant risks and consequences, highlighting the need for 


robust systems that ensure compliance and promote 


accountability. Human services encompass a wide range of 
sectors, including healthcare, social welfare, child protection, 
and criminal justice. Monitoring in these areas is essential to 
safeguard the well-being and rights of individuals, prevent 


abuses, and ensure the effective delivery of services. Here are 
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some potential risks and consequences of inadequate 


monitoring: 


Abuse and neglect: Without proper monitoring, vulnerable 
individuals may be at a higher risk of abuse, neglect, or 
exploitation. For instance, in healthcare settings, inadequate 
monitoring can lead to medical errors, mistreatment of patients, 
or substandard care. Similarly, in child protection services, 
insufficient monitoring can result in children remaining in 


abusive or neglectful environments. 


Violation of rights: Inadequate monitoring can lead to 


violations of individuals’ rights, including their civil liberties, 


privacy, and dignity. For example, in criminal justice systems, 


inadequate monitoring can result in wrongful convictions, 
excessive use of force, or violations of prisoners' rights. In 


social welfare programs, lack of monitoring can lead to 
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discrimination, improper denial of benefits, or infringement of 


recipients’ rights. 


Inefficiency and ineffective service delivery: Monitoring is 
crucial for evaluating the effectiveness and efficiency of human 
services. Without robust monitoring systems, it becomes 
challenging to identify gaps, assess performance, and make 
informed decisions for improvement. Inadequate monitoring 
may lead to wastage of resources, duplication of efforts, or the 
continuation of ineffective programs that fail to meet the needs 
of the intended beneficiaries. This is where risk assessment 
rules and key indicator rules play an important role in 


increasing the effectiveness and efficiency of the monitoring 


process by utilizing a more differential monitoring approach. 


Lack of accountability: Monitoring plays a vital role in ensuring 
accountability within human service systems. It helps identify 


and address instances of misconduct, malpractice, or non- 
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compliance with regulations and standards. Inadequate 
monitoring can result in a lack of transparency and 
accountability, allowing misconduct to go _ unnoticed, 
perpetrators to go unpunished, and systemic problems to 


persist. 


Loss of public trust: Inadequate monitoring erodes public trust 
in human service systems. When people perceive that their 
well-being, rights, or safety are compromised due to poor 
monitoring, it undermines their confidence in these services. 
Public trust is crucial for the effective functioning of human 
services, as it promotes cooperation, engagement, and 


participation of individuals and communities. 


To mitigate these risks and consequences, robust monitoring 


systems are essential. Such systems should include clear 


guidelines, regular inspections, audits, reporting mechanisms, 


and independent oversight bodies. They should also leverage 
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technology and data analysis to enhance monitoring capabilities 
and identify patterns or anomalies. Additionally, staff training 
on monitoring protocols and the establishment of a culture of 
accountability are crucial components of an_ effective 


monitoring framework. 


Inadequate monitoring in human services poses significant risks 
and consequences. It can lead to abuse, neglect, rights 
violations, inefficiencies, lack of accountability, and loss of 
public trust. Robust monitoring systems, incorporating clear 
guidelines, regular inspections, technology, and independent 
oversight, are necessary to ensure compliance, protect 
individuals, and promote accountability within human service 


sectors. 


The integration of measurement and monitoring systems into 


the licensing process in human services is a_ crucial 


development that leverages technology and data analytics to 
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track, evaluate, and verify compliance with licensing standards. 
These systems provide real-time monitoring capabilities, 
enabling early detection of non-compliance, improved 
transparency, and enhanced accountability. Let's delve into the 


details of how these systems work and the benefits they bring. 


Measurement and monitoring systems in the context of human 
services licensing involve the use of advanced technologies, 
such as sensors, cameras, electronic record-keeping systems, 
and data analytics tools. These technologies are integrated into 
the licensing process to collect, analyze, and interpret relevant 
data in real-time. The aim is to ensure that organizations and 
individuals providing human services comply with the 


established licensing standards and regulations. 


One significant advantage of integrating measurement and 


monitoring systems is the early detection of non-compliance. 


With real-time monitoring, regulatory agencies can identify 
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potential violations promptly. For example, if a human services 
facility is required to maintain a specific temperature range, 
sensors can continuously monitor the temperature levels. If 
there is a deviation from the acceptable range, an alert can be 
triggered, enabling swift corrective action. This early detection 
mechanism helps prevent potential risks and harm to 


individuals receiving those services. 


Moreover, these systems improve transparency by providing 


accurate and objective data. Instead of relying solely on 


periodic inspections or self-reported information, regulatory 


agencies can access real-time data collected by the monitoring 
systems. This data-driven approach ensures a more 
comprehensive and accurate assessment of compliance with 
licensing standards. It reduces the reliance on subjective 
observations and minimizes the possibility of information gaps 


or bias. 
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Furthermore, integrating measurement and monitoring systems 
enhances accountability for organizations and individuals 
providing human services. By continuously monitoring and 
recording data, these systems create an audit trail that can be 
used for accountability purposes. The collected data provides 
evidence of compliance or non-compliance with licensing 
standards, which can be used in regulatory investigations or 
legal proceedings if necessary. This level of accountability 
fosters a culture of responsibility and incentivizes compliance 


with licensing requirements. 


The benefits of these systems extend beyond regulatory 
agencies. Service providers themselves can benefit from real- 
time monitoring by gaining insights into their own operations 
and performance. By analyzing the data collected, they can 
identify areas for improvement, optimize resource allocation, 


and make evidence-based decisions to enhance the quality of 


their services. This data-driven approach supports continuous 
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improvement and helps providers meet and exceed licensing 


standards. 


The integration of measurement and monitoring systems into 
the licensing process in human services offers significant 
advantages. It leverages technology and data analytics to enable 
real-time monitoring, early detection of non-compliance, 
improved transparency, and enhanced accountability. These 
systems provide regulatory agencies with objective data to 
ensure compliance with licensing standards and promote the 
safety and well-being of individuals receiving human services. 
Simultaneously, service providers benefit from insights gained 


through data analysis, allowing them to optimize their 


operations and deliver higher quality services. 


Licensing measurement and monitoring systems in human 
services play a crucial role in ensuring compliance with 


regulations, tracking licensing activities, and monitoring the 
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quality and safety of services provided. These systems typically 
consist of several key components that work together to enable 
effective measurement and monitoring. Here are the main 


components: 


Comprehensive Databases: A central database is essential for 
storing all licensing-related information, including provider 
details, facility data, licensing standards, inspection reports, and 
compliance history. These databases provide a foundation for 


data collection, analysis, and reporting. 


Example: The Child Care Licensing System (CCLS) developed 


by the Administration for Children and Families in the United 


States is a comprehensive database that tracks and manages 
child care licensing information. It allows agencies to manage 


licensing processes, track violations, and generate reports. 
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Automated Data Collection Tools: Automation tools streamline 
the process of data collection by capturing information 
electronically, reducing manual effort, and improving accuracy. 
These tools can include online application forms, electronic 


submission of documentation, and automated notifications. 


Example: The Integrated Regulatory Information System 
(IRIS) used by the California Department of Social Services 
enables online application submissions, digital document 
management, and automated notifications for licensing updates. 


It simplifies the data collection process and enhances 


efficiency. 


Risk Assessment Algorithms: Risk assessment algorithms help 
identify high-risk facilities or providers that require increased 
monitoring or intervention. These algorithms analyze various 


factors such as compliance history, complaint data, inspection 
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results, and other relevant indicators to prioritize resources 


effectively. 


Example: The Risk Assessment and Management Tool (RAM) 
implemented by the Australian Government's Department of 
Health is used to assess and manage risks associated with aged 
care services. RAM employs algorithms that analyze data on 
quality indicators, complaints, and non-compliance to 


determine risk levels and allocate resources accordingly. 


Data Visualization Platforms: Data visualization platforms 


present licensing data in a user-friendly and meaningful way, 


allowing regulatory agencies to monitor trends, identify 


patterns, and make data-driven decisions. These platforms often 


include interactive dashboards, charts, and reports. 


Example: The Licensing Information System (LIS) developed 


by the Department of Health and Human Services in the state 


Licensing Measurement & Monitoring Systems Fiene 


of Maine provides a data visualization platform that allows 
users to generate customized reports, view interactive charts, 


and track licensing compliance trends. 


Compliance Monitoring Tools: Compliance monitoring tools 
assist in conducting inspections, audits, and other monitoring 
activities efficiently. These tools can include mobile 
applications for inspectors to collect data on-site, electronic 


checklists, and automated scheduling of inspections. 


Example: The Licensing Automation System (LAS) 
implemented by the Minnesota Department of Human Services 
offers mobile applications for licensing staff to perform 


inspections, record findings, and generate inspection reports on 


the go. It simplifies the monitoring process and improves 


accuracy. 
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Overall, these components work together to create effective 
licensing measurement and monitoring systems in human 
services. By leveraging comprehensive databases, automated 
data collection tools, risk assessment algorithms, data 
visualization platforms, and compliance monitoring tools, 
regulatory agencies can enhance their oversight capabilities, 
improve efficiency, and ensure the provision of high-quality 


services while maintaining compliance with regulations. 


Licensing measurement and monitoring systems have had a 
significant impact on regulatory administration and the human 


services sector. These systems play a crucial role in enabling 


regulators to proactively identify potential risks, address 


compliance issues promptly, and ensure the safety and quality 
of services provided. In this response, we will discuss the 
impact of these systems and provide case studies and examples 
that illustrate the positive outcomes achieved through their 


implementation. 


Licensing Measurement & Monitoring Systems 


One of the primary benefits of licensing measurement and 
monitoring systems is their ability to provide regulators with 
real-time data and insights. These systems collect and analyze 
various metrics and indicators, allowing regulators to monitor 
the performance and compliance of service providers. By 
having access to accurate and up-to-date information, 
regulators can proactively identify potential risks and address 


them before they escalate into serious problems. 


For instance, let's consider the case of a regulatory agency 
responsible for overseeing childcare facilities. By 
implementing a licensing measurement and monitoring system, 
the agency can track key indicators such as staff-to-child ratios, 
health and safety inspections, and educational programs. If the 


system detects any deviations from the established standards, it 


can alert regulators, enabling them to intervene promptly. This 
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proactive approach helps prevent incidents and ensures that 


children receive appropriate care and support. 


Another positive outcome of licensing measurement and 
monitoring systems is improved compliance management. 
These systems streamline the process of monitoring and 
assessing compliance with regulations and standards. Service 
providers can input data directly into the system, reducing the 
administrative burden and ensuring accuracy. Regulators can 
then use this data to identify patterns, assess compliance levels, 


and take appropriate actions if non-compliance is detected. 


For example, let's consider the case of a regulatory agency 
overseeing healthcare facilities. With a licensing measurement 
and monitoring system in place, the agency can track indicators 
such as medication errors, infection rates, and patient 


satisfaction scores. If the system identifies a healthcare facility 


with consistently high medication error rates, regulators can 
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conduct targeted inspections and work closely with the facility 
to implement corrective measures. This proactive approach not 
only improves patient safety but also helps service providers 


enhance the quality of care they deliver. 


Furthermore, licensing measurement and monitoring systems 
contribute to transparency and accountability in the human 
services sector. These systems provide a centralized platform 
where regulators, service providers, and the public can access 
information about licensing status, compliance records, and 
performance metrics. By promoting transparency, these 
systems help build trust among stakeholders and empower 


individuals to make informed decisions about service providers. 


For instance, in the context of elder care services, a licensing 
measurement and monitoring system can provide a public 


database that includes information on the licensing status of 


assisted living facilities, compliance records related to safety 
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standards, and ratings based on resident satisfaction surveys. 
This enables families and individuals seeking care for their 
loved ones to make informed choices and select facilities that 


meet their specific needs. 


Licensing measurement and monitoring systems have had a 
transformative impact on regulatory administration and the 
human services sector. These systems enable regulators to 
proactively identify potential risks, address compliance issues 
promptly, and ensure the safety and quality of services 


provided. Through case studies and examples, we have seen 


how these systems have improved oversight in childcare, 


healthcare, and elder care, leading to positive outcomes such as 
enhanced safety, improved compliance, and _ increased 
transparency. The implementation of such systems has the 
potential to further strengthen regulatory efforts and promote 


the well-being of individuals receiving human services. 
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Licensing measurement and monitoring systems can present 
various challenges and considerations, including privacy 
concerns, data security, resource constraints, and the need for 
ongoing system updates and maintenance. Addressing these 
challenges is crucial to ensure the effective implementation and 
operation of these systems. Additionally, collaboration between 
regulatory agencies, stakeholders, and technology providers is 
essential to overcome these challenges and maximize the 


benefits of these systems. 


Privacy concerns: Measurement and monitoring systems often 


involve the collection and analysis of sensitive data, such as 


personal information or proprietary business data. It is 


important to establish robust privacy policies and legal 
frameworks to protect individuals’ privacy rights and ensure 
compliance with relevant data protection regulations. 


Implementing anonymization techniques, data minimization 
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principles, and obtaining appropriate consent can help mitigate 


privacy concerns. 


Data security: The storage, transmission, and analysis of 
measurement and monitoring data require robust security 


measures to prevent unauthorized access, data breaches, or 


cyber-attacks. Encryption, access controls, regular security 


audits, and adherence to industry best practices can help 


safeguard the data and maintain its integrity and confidentiality. 


Resource constraints: Licensing measurement and monitoring 
systems can pose financial and logistical challenges, 
particularly for smaller organizations or developing countries 
with limited resources. These systems may require substantial 
investments in infrastructure, equipment, and skilled personnel. 
Adequate funding mechanisms, public-private partnerships, 
and capacity-building initiatives can help address resource 


constraints and ensure broader access to these systems. 
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Ongoing system updates and maintenance: Measurement and 
monitoring systems must be regularly updated to keep pace 
with evolving technologies, regulatory requirements, and 
scientific advancements. This necessitates ongoing 
maintenance, software updates, calibration, and quality control 
procedures. Collaboration between regulatory agencies, 
technology providers, and stakeholders is crucial to establish 
effective mechanisms for system maintenance, ensuring that the 


systems remain accurate, reliable, and up-to-date. 


Collaboration between regulatory agencies, stakeholders, and 


technology providers: Overcoming the challenges associated 
with licensing measurement and monitoring systems requires a 
collaborative approach. Regulatory agencies should engage in 
constructive dialogues with stakeholders, including industry 
representatives, environmental organizations, and community 


groups. Collaboration can help address concerns, establish 


Licensing Measurement & Monitoring Systems Fiene 


common standards, and promote’ transparency and 
accountability. Technology providers can contribute by 
developing user-friendly and interoperable systems that meet 
regulatory requirements while minimizing the burden on end- 


users. 


Collaboration among regulatory agencies, stakeholders, and 
technology providers is critical to ensure the successful 
implementation of measurement and monitoring systems. By 
working together, these entities can develop robust policies, 
address privacy concerns, enhance data security, allocate 
necessary resources, and establish mechanisms for ongoing 
system updates and maintenance. This collaborative approach 
will maximize the effectiveness of these systems in monitoring 


and safeguarding various aspects of public health, 


environmental quality, and regulatory compliance. 
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Licensing measurement and monitoring systems play a crucial 
role in human services regulatory administration by ensuring 
compliance, enhancing service quality, and _ protecting 
individuals receiving care. Integrating regulatory science 
principles into licensing processes further strengthens these 


benefits. 


One significant aspect of licensing measurement and 
monitoring systems is their ability to promote compliance. 
These systems provide a standardized framework for evaluating 
and assessing the compliance of service providers with 
established regulations and standards. By implementing these 
systems, regulatory authorities can systematically track and 
measure compliance levels, identify areas of non-compliance, 


and take appropriate actions to rectify any deficiencies. This 


helps maintain a high level of accountability among service 


providers, ensuring they adhere to the required standards and 


regulations. 
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Moreover, integrating regulatory science principles into 
licensing processes brings several advantages. Regulatory 
science applies scientific knowledge and methodologies to 
inform regulatory decision-making. By incorporating these 
principles into licensing, regulators can leverage evidence- 
based approaches to establish standards, design measurement 
tools, and set performance benchmarks. This approach 
promotes objectivity, transparency, and consistency in the 
licensing process, ensuring that decisions are based on sound 


scientific evidence rather than subjective judgment. 


Another key benefit is the potential for improved service 
quality. Licensing measurement and monitoring systems enable 


regulators to gather comprehensive data on service providers’ 


performance, outcomes, and service quality indicators. This 


information allows for a thorough assessment of service 


delivery, identifying strengths and weaknesses in the system. 
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By analyzing this data, regulators can provide feedback, 
guidance, and support to service providers, fostering continuous 
improvement in service quality. This leads to better outcomes 
for individuals receiving care and enhances overall service 


provision within the human services sector. 


Furthermore, licensing measurement and monitoring systems 
are instrumental in protecting the well-being of individuals 
receiving care. These systems help identify potential risks, such 
as violations of safety protocols or instances of abuse or neglect. 
By closely monitoring service providers, regulators can swiftly 
respond to any issues, take necessary corrective actions, and 
ensure the safety and well-being of vulnerable populations. 


Regular monitoring also acts as a deterrent, encouraging service 


providers to maintain high standards and comply with 


regulations to avoid penalties or sanctions. 
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Looking ahead, the field of regulatory science and measurement 
and monitoring systems is continually evolving. Advances in 
technology, data analytics, and artificial intelligence present 
opportunities for further advancements in these systems. For 
example, the integration of real-time data collection and 
analysis can enhance the effectiveness and efficiency of 
monitoring processes. Predictive analytics and risk assessment 
models can help regulators proactively identify potential areas 
of concern and allocate resources accordingly. Additionally, the 
incorporation of feedback from individuals receiving care and 
other stakeholders can further refine measurement systems, 


ensuring they capture the most relevant and meaningful 


indicators of service quality. 


In conclusion, licensing measurement and monitoring systems 
are vital components of human _ services regulatory 
administration. By integrating regulatory science principles, 


these systems promote compliance, improve service quality, 
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and protect individuals receiving care. As regulatory science 
continues to evolve, the potential for further advancements in 


measurement and monitoring systems is promising, enabling 


regulators to better fulfill their mandate of safeguarding the 


well-being of vulnerable populations. 


Licensing Measurement & Monitoring Systems 


Importance of the Theory of Regulatory Compliance 


Introduction 


Regulatory compliance refers to the process by which 
individuals, organizations, or entities adhere to and fulfill the 
requirements set forth by relevant laws, regulations, and 
industry standards. It involves ensuring that policies, 
procedures, and practices align with the specific legal and 


regulatory frameworks applicable to a particular industry or 


jurisdiction. 


Compliance involves actively identifying and understanding 
the relevant regulations, establishing internal controls and 
processes to meet those requirements, and consistently 
monitoring and reviewing operations to ensure ongoing 
adherence. It encompasses various aspects, such as legal, 


financial, operational, and ethical considerations, and aims to 
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ensure that organizations operate within the boundaries of the 
law, maintain ethical standards, and fulfill their responsibilities 


to stakeholders, customers, and the public. 


The theory of regulatory compliance provides a framework for 
understanding the underlying principles and concepts that guide 
the compliance process. It encompasses several key elements 
that shape the approach to achieving and maintaining 
compliance. Here is an overview of the theory of regulatory 


compliance: 


Legal and Regulatory Environment: The theory recognizes that 


regulatory compliance is rooted in the legal and regulatory 


landscape. It acknowledges the importance of identifying and 
understanding applicable laws, regulations, and standards that 


govern an industry or jurisdiction. 
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Risk Management: The theory emphasizes the proactive 
identification, assessment, and management of risks associated 
with non-compliance. It highlights the need to establish robust 
risk management processes to mitigate legal, financial, 


operational, and reputational risks. 


Policies and Procedures: Effective compliance requires the 
development and implementation of comprehensive policies 
and procedures. The theory underscores the significance of 
clear, well-documented, and communicated policies that guide 


employees in adhering to regulatory requirements. 


Internal Controls: The theory emphasizes the establishment of 


internal controls to ensure compliance. This involves designing 


and implementing systems, processes, and checks that monitor 
and mitigate risks, detect and prevent non-compliance, and 


promote accountability. 
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Training and Awareness: Recognizing the role of individuals in 
compliance, the theory highlights the importance of training 
programs and awareness initiatives. It emphasizes educating 
employees about applicable regulations, ethical standards, and 


the organization's compliance obligations. 


Monitoring and Auditing: The theory acknowledges the need 
for ongoing monitoring and auditing to assess compliance 
effectiveness. Regular internal audits, reviews, and assessments 


help identify gaps, weaknesses, and areas for improvement, 


ensuring continuous compliance efforts. 


Reporting and Documentation: The theory stresses the 
significance of accurate and timely reporting of compliance 
activities. It underscores the need to maintain proper 
documentation, records, and evidence of compliance processes, 


actions taken, and outcomes achieved. 
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Compliance Culture: The theory recognizes that compliance is 
not solely a set of rules and processes but also a cultural 
mindset. It highlights the importance of fostering a culture of 
compliance within an organization, where integrity, ethics, and 
adherence to regulations are valued and embedded in the 


organizational DNA. 


Accountability and Enforcement: The theory acknowledges 
that compliance requires accountability for non-compliance. It 
recognizes the role of regulatory bodies, internal enforcement 
mechanisms, and disciplinary actions in promoting compliance 


and deterring violations. 


Continuous Improvement: Finally, the theory emphasizes the 
need for continuous improvement in compliance efforts. It 


encourages organizations to learn from past experiences, adapt 


to evolving regulations, embrace emerging best practices, and 


strive for excellence in their compliance initiatives. 
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By understanding and applying the theory of regulatory 
compliance, organizations can establish a solid foundation for 
effective compliance management, minimize risks, and uphold 


legal and ethical standards in their operations. 


Ensuring Legal and Ethical Practices 


Compliance with laws and regulations is a fundamental aspect 
of the theory of regulatory compliance. It recognizes that 
adherence to legal and regulatory requirements is crucial for 


organizations to operate within the boundaries set by governing 


bodies and to fulfill their obligations to stakeholders. Here are 


key points related to compliance with laws and regulations in 


the context of the theory of regulatory compliance: 


Understanding Applicable Laws: The theory emphasizes the 


importance of identifying and comprehending the specific laws 
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and regulations that pertain to an organization's industry, 
jurisdiction, and operational activities. This involves staying 
updated with changes in regulations and interpreting their 


implications for the organization. 


Regulatory Research and Analysis: Organizations need to 
conduct thorough research and analysis to determine how laws 
and regulations apply to their operations. This includes 
examining regulatory frameworks, guidance documents, legal 


precedents, and industry-specific requirements. 


Compliance Obligations: The theory recognizes _ that 
compliance obligations vary based on the nature of the 
organization's activities. It stresses the need to determine the 
specific requirements, obligations, and standards that the 


organization must meet to ensure legal and regulatory 


compliance. 
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Compliance Program Development: To achieve compliance, 
the theory highlights the importance of developing a 
comprehensive compliance program tailored to the 
organization's needs. This involves establishing policies, 
procedures, and controls that align with legal and regulatory 


requirements. 


Regulatory Reporting and Filings: Compliance entails fulfilling 
reporting obligations to regulatory authorities. The theory 
emphasizes the significance of timely and accurate reporting, 


including financial statements, disclosures, permits, licenses, 


certifications, and other regulatory filings. 


Compliance Monitoring and Auditing: The theory underscores 
the need for ongoing monitoring and auditing of compliance 
efforts. Regular reviews help identify potential compliance 
gaps, assess the effectiveness of controls, and ensure corrective 


actions are taken to address non-compliance. 
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Compliance Documentation: Documentation plays a critical 
role in compliance. The theory highlights the importance of 
maintaining accurate and comprehensive records of compliance 
activities, including policies, procedures, training materials, 


audit reports, incident reports, and evidence of compliance. 


Compliance Risk Assessment: Organizations should conduct 
compliance risk assessments to identify and evaluate potential 
risks associated with non-compliance. This allows for the 


implementation of risk mitigation strategies, such as internal 


controls, training programs, and monitoring systems. 


Enforcement and Consequences: The theory acknowledges that 
non-compliance can lead to legal and financial consequences. 
It emphasizes the need for organizations to understand the 
potential penalties, fines, sanctions, and reputational damage 


that can result from violations of laws and regulations. 
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Regulatory Engagement and Communication: Organizations 
should actively engage with regulatory authorities and maintain 
open lines of communication. The theory emphasizes the 
importance of understanding regulatory expectations, seeking 
guidance when needed, and participating in industry 


consultations. 


By emphasizing compliance with laws and regulations, the 
theory of regulatory compliance aims to ensure that 


organizations operate within legal boundaries, mitigate risks, 


protect stakeholders, and maintain a strong ethical foundation 


in their operations. 


Protection of Consumers and Public Interest 


Protection of consumers and the public interest is a fundamental 


objective of regulatory compliance. Regulatory compliance 
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refers to the adherence of individuals, organizations, or 
businesses to laws, regulations, and guidelines set forth by 
governing bodies or regulatory authorities. It aims to ensure that 
entities operate in a manner that safeguards the interests of 


consumers and the general public. 


The theory behind regulatory compliance is rooted in the belief 
that certain industries or activities require oversight and 
regulation to prevent harm, ensure fair competition, and 


maintain public trust. By establishing rules and standards, 


regulatory bodies seek to create a level playing field, promote 


transparency, and protect the well-being of consumers. 


Key principles and considerations associated with regulatory 
compliance for the protection of consumers and public interest 


include: 


Licensing Measurement & Monitoring Systems Fiene 


Consumer Protection: Regulatory compliance frameworks 
typically include provisions to safeguard consumers from 
fraudulent, deceptive, or unfair practices. This involves 
regulations related to product safety, labeling, advertising, 


pricing, warranties, and consumer rights. 


Public Health and Safety: Compliance regulations often address 
public health and safety concerns. For instance, in the 
pharmaceutical industry, compliance with drug safety 
regulations ensures that medications meet quality standards and 


do not pose unreasonable risks to patients. 


Market Integrity: Regulatory compliance helps maintain the 
integrity of markets by prohibiting anti-competitive behavior, 
ensuring fair trading practices, and preventing market 
manipulation or insider trading. These regulations promote fair 


competition and protect consumers from monopolistic 


practices. 
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Data Protection and Privacy: With the increasing prevalence of 
data-driven technologies, regulatory compliance frameworks 
emphasize the protection of personal information and privacy 
rights. Regulations like the European Union's General Data 
Protection Regulation (GDPR) aim to safeguard consumer data 
and establish guidelines for its lawful collection, storage, and 


use. 


Financial Stability: Regulatory compliance plays a crucial role 
in the financial sector to prevent fraud, money laundering, and 
unethical practices that can destabilize markets or harm 
consumers. Regulations impose standards for capital adequacy, 


risk management, disclosure, and consumer financial 


protection. 


Ethical Considerations: Compliance regulations often 


incorporate ethical considerations to ensure responsible and 
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ethical behavior by individuals and organizations. This may 
involve guidelines on corporate governance, social 


responsibility, environmental sustainability, or labor practices. 


To ensure effective regulatory compliance, regulatory bodies 
conduct inspections, audits, and enforcement actions. Non- 
compliance can result in penalties, fines, or legal actions against 
the offending parties. Moreover, compliance management 
systems, internal controls, and self-regulatory mechanisms are 


employed by organizations to proactively adhere to regulatory 


requirements and promote a culture of compliance. 


Overall, the theory of regulatory compliance revolves around 
the idea that by setting and enforcing rules, regulators can 
protect consumers, preserve public interest, and maintain the 


stability and fairness of various sectors in society. 
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Financial Stability and Risk Management 


Financial stability and risk management are critical components 
of regulatory compliance. The theory of regulatory compliance 
emphasizes the importance of establishing and enforcing 
regulations to ensure the stability and integrity of financial 


systems, protect consumers, and mitigate systemic risks. 


Here are some key aspects of the theory of regulatory 


compliance related to financial stability and risk management: 


Prudential Regulation: Prudential regulation focuses on 
ensuring the soundness and stability of financial institutions, 


such as banks, insurance companies, and investment firms. 


Regulatory compliance frameworks impose requirements 


related to capital adequacy, risk management, liquidity, and 
asset quality to prevent excessive risk-taking and protect the 


financial system from disruptions. 
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Systemic Risk Mitigation: Regulatory compliance measures 
aim to identify and mitigate systemic risks that can have 
widespread adverse effects on the financial system. This 
includes regulations on risk concentration, interconnectedness, 
and exposure limits to prevent the domino effect of failures and 


contagion across institutions. 


Risk Assessment and Monitoring: Regulatory compliance 
frameworks often require financial institutions to conduct 
thorough risk assessments and implement robust risk 
management practices. This involves identifying, measuring, 
and monitoring various types of risks, including credit risk, 


market risk, liquidity risk, and operational risk. Compliance 


regulations may prescribe specific methodologies, reporting 


requirements, and stress testing to ensure that risks are 


adequately identified and managed. 
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Transparency and Disclosure: Regulatory compliance promotes 
transparency in financial markets by requiring financial 
institutions to provide accurate and timely disclosure of 
relevant information to investors, regulators, and the public. 
This includes financial reporting, disclosures of risk exposures, 
and information about the institution's financial health. 
Transparent reporting helps stakeholders make informed 
decisions, enhances market efficiency, and fosters trust in the 


financial system. 


Consumer Financial Protection: Regulatory compliance 
frameworks incorporate measures to protect consumers in 
financial transactions. This includes regulations on fair lending 
practices, disclosure requirements for financial products and 
services, and regulations against abusive or predatory practices. 
These regulations aim to ensure that consumers are treated 


fairly, have access to transparent information, and are protected 


from fraudulent or deceptive practices. 
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Regulatory Oversight and Enforcement: Regulatory 
compliance is reinforced by regulatory bodies that oversee 
financial institutions, enforce compliance, and impose penalties 
for non-compliance. These regulatory authorities monitor 
institutions’ compliance with regulations, conduct audits and 
examinations, and take enforcement actions when violations are 
identified. Such oversight ensures accountability and promotes 


a culture of compliance within the financial industry. 


By adhering to regulatory compliance requirements, financial 
institutions are expected to minimize risks, enhance stability, 
and maintain the confidence of investors and the public. 
Compliance management systems, internal controls, and risk 


management frameworks are utilized by financial institutions to 


meet regulatory obligations and proactively manage risks. 
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Overall, the theory of regulatory compliance underscores the 
role of regulations in promoting financial stability, mitigating 
risks, protecting consumers, and maintaining the integrity of 
financial systems. Compliance with these regulations helps 
build a resilient financial sector that can withstand shocks and 


contribute to overall economic stability. 


Preserving Competitive Market Environment 


Preserving a competitive market environment is essential for 
fostering innovation, encouraging efficiency, and benefiting 
consumers. The theory of regulatory compliance is closely 
linked to this objective, as it involves establishing and enforcing 
rules and regulations that promote fair competition and prevent 


anti-competitive practices. 


The theory of regulatory compliance is based on the idea that 


regulatory frameworks can help create a level playing field for 
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all market participants. By setting clear rules and standards, 
regulators aim to ensure that businesses operate within the 
bounds of fair competition. Compliance with these regulations 
helps prevent monopolistic behavior, collusion, price-fixing, 


and other practices that could harm competition. 


Here are a few key principles related to preserving a 
competitive market environment and the theory of regulatory 


compliance: 


Anti-Trust Laws: Anti-trust laws are designed to promote 


competition by preventing the abuse of market power. They 


prohibit practices such as monopolies, cartels, price-fixing, and 


mergers that may substantially lessen competition. Regulators 
enforce these laws to preserve a competitive landscape and 


protect consumer interests. 
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Market Entry and Exit: Regulatory frameworks should facilitate 
the entry of new businesses into the market while allowing 
existing ones to exit if they are unable to compete effectively. 
Barriers to entry, such as excessive licensing requirements or 
unfair regulations, can hinder competition. Regulatory 
compliance should aim to reduce these barriers and ensure fair 


access for all participants. 


Consumer Protection: A competitive market environment 
should prioritize consumer welfare. Regulatory compliance 
plays a crucial role in safeguarding consumer interests by 
ensuring transparency, fair pricing, quality standards, and 


adequate information disclosure. Consumer protection laws and 


regulations address issues such as misleading advertising, 


product safety, and fair dispute resolution mechanisms. 


Enforcement and Monitoring: Regulatory agencies are 


responsible for enforcing compliance with regulations. They 
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monitor market activities, investigate potential violations, and 
take appropriate enforcement actions when necessary. Effective 
enforcement requires sufficient resources, expertise, and 
collaboration among regulators, ensuring a level playing field 


for all participants. 


International Cooperation: In a globalized economy, preserving 
a competitive market environment requires international 
cooperation. Collaboration between regulatory authorities 
across jurisdictions can help address cross-border anti- 


competitive practices, harmonize regulatory standards, and 


promote fair competition in the global marketplace. 


Overall, the theory of regulatory compliance supports the 
notion that well-designed and effectively enforced regulations 
can foster a competitive market environment. By promoting fair 


competition, preventing anti-competitive practices, and 
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protecting consumer interests, regulatory compliance 


contributes to a healthy and vibrant marketplace. 


Establishing Trust and Credibility 


Establishing trust and credibility is crucial for regulatory 
compliance efforts. The theory of regulatory compliance 
recognizes that trust is essential in fostering cooperation 
between regulatory authorities, businesses, and _ other 
stakeholders. Trust is built when regulations are transparent, 


consistently enforced, and perceived as fair and unbiased. 


Here are some key aspects of establishing trust and credibility 


in the context of regulatory compliance: 


Transparency: Transparency is a fundamental principle in 


regulatory compliance. Regulations and their enforcement 


processes should be clearly communicated and accessible to all 
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stakeholders. Openness helps build trust by ensuring that the 
rules are known and understood by businesses and individuals, 


reducing uncertainty and promoting voluntary compliance. 


Consistency: Consistency in applying regulations is critical for 
building trust. Regulators should strive to enforce regulations 
uniformly and without favoritism or discrimination. Consistent 
enforcement establishes a level playing field, fostering trust 
among market participants who know that everyone is subject 


to the same rules. 


Accountability: Regulatory authorities should be accountable 
for their actions. This includes being transparent about 


decision-making processes, justifying regulatory actions, and 


providing avenues for recourse and appeal. Accountability 


mechanisms help prevent abuse of regulatory power and build 


trust by demonstrating fairness and impartiality. 
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Collaboration and Engagement: Regulatory compliance efforts 
benefit from collaboration and engagement with various 
stakeholders. This includes businesses, industry associations, 
consumer groups, and experts. Involving stakeholders in the 
regulatory process helps ensure that regulations are practical, 
effective, and well-understood. Collaboration also enhances 
trust by incorporating diverse perspectives and building 


consensus. 


Risk-Based Approach: A risk-based approach to regulation can 


contribute to trust and credibility. It involves assessing risks, 


prioritizing enforcement efforts based on the potential harm to 


the public or the market, and proportionately allocating 
regulatory resources. This approach demonstrates that 
regulatory actions are driven by objective evaluations and the 
need to address significant risks, enhancing trust in the 


regulatory system. 
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Continuous Improvement: Regulatory compliance should be a 
dynamic and evolving process. Regular evaluation and 
improvement of regulations and enforcement mechanisms are 
essential for maintaining trust and credibility. Regulators 
should engage in periodic reviews, solicit feedback from 
stakeholders, and adapt regulations to changing market 


dynamics and emerging challenges. 


Effective Communication: Clear and effective communication 
is vital for establishing trust. Regulators should communicate 
expectations, obligations, and changes in regulations in a timely 
and accessible manner. Communication channels should be 


open to addressing queries, providing guidance, and clarifying 


regulatory requirements, fostering trust by ensuring 


transparency and promoting compliance. 


In summary, trust and credibility are foundational elements of 


successful regulatory compliance. By promoting transparency, 
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consistency, accountability, collaboration, and _ effective 
communication, regulatory authorities can establish a trusted 
regulatory framework that fosters compliance and cooperation 


among stakeholders. 


Penalties and Consequences of Non-Compliance 


Regulatory compliance refers to the act of adhering to laws, 
regulations, guidelines, and standards set forth by governing 
bodies or regulatory agencies. Non-compliance occurs when 
individuals, organizations, or businesses fail to meet these 
requirements. The penalties and consequences of non- 


compliance can vary depending on the specific regulations and 


jurisdictions involved. Here are some common penalties and 


consequences: 


Fines and Monetary Penalties: Regulatory agencies often have 


the authority to impose fines and monetary penalties for non- 
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compliance. The amount of the penalty may vary depending on 
the severity of the violation and the regulatory framework in 
place. These fines can be substantial and can significantly 


impact the finances of non-compliant entities. 


Legal Proceedings and Lawsuits: Non-compliance may lead to 
legal action, including lawsuits filed by affected parties or 
regulatory bodies. This can result in costly litigation, potential 


damages, and a tarnished reputation. 


License Revocation or Suspension: Certain industries and 


professions require licenses or permits to operate legally. Non- 


compliance can lead to the revocation or suspension of these 
licenses, effectively shutting down the business or preventing 


individuals from practicing their profession. 


Regulatory Audits and Inspections: Regulatory agencies may 


conduct audits and inspections to assess compliance. Non- 
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compliant entities may face increased scrutiny, additional 
audits, or more frequent inspections, leading to disruption of 


operations and additional costs. 


Reputational Damage: Non-compliance can harm an 
organization's reputation, leading to loss of customer trust, 
decreased sales, and difficulty attracting new customers. 
Negative publicity and media attention can have long-lasting 


effects on brand value and perception. 


Corrective Actions and Remediation Costs: In many cases, non- 
compliant entities are required to take corrective actions to 


address the violations. This may involve implementing new 


policies, procedures, or systems, as well as investing in training 


and education. The costs associated with these remediation 


efforts can be significant. 
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Criminal Charges and Penalties: In cases of serious non- 
compliance, intentional violations, or fraudulent activities, 
criminal charges may be pursued. This can result in fines, 


imprisonment, or both, depending on the severity of the offense. 


The theory of regulatory compliance seeks to understand why 
individuals or organizations choose to comply or not comply 
with regulations. Factors influencing compliance behavior 
include perceived legitimacy of regulations, trust in regulatory 
agencies, the presence of effective enforcement mechanisms, 
and the perceived costs and benefits of compliance. The theory 


emphasizes the importance of clear communication, consistent 


enforcement, and proportionate penalties to achieve higher 


compliance rates. 
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Compliance Programs and Frameworks 


Compliance programs and frameworks are designed to help 
organizations establish and maintain a culture of regulatory 
compliance. They provide a_ structured approach to 


understanding and meeting regulatory requirements, mitigating 


risks, and promoting ethical behavior. Additionally, 


compliance programs help organizations detect and address 


non-compliance issues promptly and effectively. 


Here are some common compliance programs and frameworks: 


Compliance Management System (CMS): A CMS is a 
comprehensive framework that encompasses _ policies, 
procedures, processes, and controls to manage compliance 
within an organization. It includes elements such as risk 
assessment, compliance training, monitoring and auditing, 


incident reporting, and corrective action planning. 
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ISO 19600: This international standard provides guidelines for 
establishing, implementing, evaluating, and improving a 
compliance management system. It emphasizes a risk-based 
approach to compliance and provides a framework for 
organizations to identify, analyze, and address their compliance 


obligations effectively. 


COSO Framework: The Committee of Sponsoring 
Organizations of the ‘Treadway Commission (COSO) 


developed a framework that focuses on internal controls and 


risk management. While not specifically geared towards 


compliance, it provides a solid foundation for managing 


compliance risks within an organization. 


Federal Sentencing Guidelines (FSG): The U.S. Federal 
Sentencing Guidelines provide guidance for organizations on 


establishing effective compliance programs. They outline 
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specific factors that organizations should consider when 
developing compliance programs, such as conducting risk 
assessments, implementing training and communication 


programs, and monitoring compliance. 


Principle-Based Approach: The principle-based approach to 
compliance focuses on establishing a set of core principles and 
values that guide an organization's compliance efforts. It 
emphasizes ethical conduct, integrity, and accountability as the 
foundation for compliance programs. This approach encourages 


employees to make ethical decisions and act in accordance with 


the organization's values. 


The theory of regulatory compliance explores the factors that 
influence compliance behavior and the effectiveness of 
compliance programs. It recognizes that compliance is not 
solely driven by the fear of penalties but also by factors such as 


organizational culture, perceived legitimacy of regulations, and 
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the presence of strong internal controls. The theory suggests 


that effective compliance programs should: 


Clearly communicate regulatory requirements and expectations 


to employees and stakeholders. 


Foster a culture of compliance by promoting ethical behavior, 


accountability, and integrity. 


Provide training and education to employees to enhance their 


understanding of compliance obligations. 


Implement monitoring and auditing mechanisms to detect and 


address non-compliance promptly. 


Establish strong internal controls and risk management 


processes to mitigate compliance risks. 
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Encourage reporting of potential compliance issues and provide 


channels for anonymous reporting. 


Continuously evaluate and improve the compliance program 


based on feedback and changes in regulations. 


By understanding the theory of regulatory compliance and 
implementing effective compliance programs, organizations 
can enhance their ability to meet regulatory requirements, 


manage risks, and uphold ethical standards. 


Role of Technology in Regulatory Compliance and monitoring 


and reporting tools 


Technology plays a crucial role in regulatory compliance by 


providing tools and systems that help organizations monitor and 


report their adherence to regulatory requirements. Here are 


some key ways technology supports regulatory compliance: 
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Automation and Workflow Management: Technology enables 
the automation of various compliance processes, such as data 
collection, analysis, and reporting. Workflow management 
systems help streamline compliance tasks by providing clear 
processes and guidelines, ensuring consistent and efficient 


execution. 


Data Management and Analysis: Compliance often involves 
handling large volumes of data. Technology solutions, such as 
data management systems and analytics tools, facilitate the 
collection, storage, organization, and analysis of data for 
compliance purposes. These systems can identify patterns, 
anomalies, and trends in the data, helping organizations detect 


and address compliance risks. 


Monitoring and Surveillance: Technology enables real-time 


monitoring and surveillance of activities, transactions, and 
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communications to identify potential compliance violations. 
Advanced monitoring tools use algorithms and machine 
learning techniques to detect suspicious behavior, fraud, market 


manipulation, or any non-compliant activities. 


Reporting and Documentation: Compliance requires accurate 
and timely reporting to regulatory authorities. Technology 
offers reporting tools that help automate the creation of 
regulatory reports, ensuring the required information is 
captured, organized, and submitted in the appropriate format. 
These tools often include templates, data mapping capabilities, 


and integration with existing systems. 


Audit Trail and Documentation Management: Technology 
allows organizations to maintain a comprehensive audit trail 


and documentation of compliance activities. Digital systems 


enable the secure storage, retrieval, and tracking of compliance- 
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related documents, making it easier to demonstrate compliance 


during audits or investigations. 


Risk Assessment and Compliance Monitoring: Technology 
supports risk assessment processes by providing tools for 
identifying, assessing, and prioritizing compliance risks. 
Compliance monitoring tools can continuously track regulatory 
changes and updates, ensuring organizations stay informed and 


adapt their compliance programs accordingly. 


Training and Education: Technology can be utilized to deliver 
compliance training and educational materials to employees 
and stakeholders. Online learning platforms, webinars, and 
interactive modules can provide accessible and engaging 
compliance training programs, ensuring widespread 


understanding of regulatory requirements and promoting a 


culture of compliance. 
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Overall, technology plays a vital role in enhancing the 
efficiency, accuracy, and effectiveness of regulatory 
compliance efforts. By leveraging technology, organizations 
can better manage compliance requirements, mitigate risks, and 
ensure adherence to regulations in an increasingly complex 


regulatory landscape. 


Conclusion Recap of the importance of the theory of regulatory 


compliance 


The theory of regulatory compliance is of great importance in 


various domains, particularly in legal and business contexts. It 


refers to the set of rules, regulations, and standards that 


individuals, organizations, and industries must follow to ensure 


compliance with applicable laws and regulations. 


Here are some key points highlighting the importance of the 


theory of regulatory compliance: 
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Legal Compliance: Regulatory compliance ensures that 
individuals and organizations adhere to laws and regulations set 
forth by governing bodies. This helps maintain law and order in 


society and promotes fairness, transparency, and accountability. 


Risk Mitigation: Compliance measures help identify and 
mitigate potential risks associated with non-compliance. By 
following regulations, organizations can minimize legal, 
financial, reputational, and operational risks. Compliance 
frameworks often include risk assessment and management 


components, enabling proactive risk mitigation. 


Consumer Protection: Compliance regulations often aim to 
protect consumers’ rights and interests. Compliance with 


consumer protection laws ensures fair business practices, 


prevents fraud, and enhances consumer trust in products and 


services. 
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Data Privacy and Security: In the digital age, data privacy and 
security have become crucial concerns. Regulatory compliance 
frameworks, such as the General Data Protection Regulation 
(GDPR), enforce strict guidelines for handling personal data. 
Compliance helps safeguard sensitive information, maintain 


privacy, and prevent data breaches. 


Ethical Standards: Compliance extends beyond legal 
obligations and encompasses ethical standards. It encourages 
organizations to adopt ethical business practices, such as fair 
competition, anti-corruption measures, and environmental 


sustainability. Compliance frameworks often incorporate 


ethical guidelines to promote responsible conduct. 


Industry Standards: Many industries have specific regulatory 
compliance requirements tailored to their unique characteristics 


and risks. Compliance with industry-specific regulations 
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ensures safety, quality, and standardization within the sector. 
Examples include regulations in healthcare, finance, energy, 


and manufacturing. 


Reputation and Trust: Compliance with regulations builds a 
positive reputation for individuals and organizations. It 
demonstrates commitment to legal and ethical standards, 
fostering trust among customers, investors, and other 
stakeholders. A strong reputation for compliance can lead to 


increased business opportunities and competitive advantage. 


Legal Consequences: Non-compliance with regulatory 
requirements can have severe legal consequences, including 
fines, penalties, sanctions, and legal liabilities. Violations can 
result in damaged reputation, loss of business licenses, and even 


criminal charges. Compliance helps organizations avoid legal 


pitfalls and maintain a good standing with regulatory 


authorities. 
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Global Business Landscape: With increasing globalization, 
organizations often need to navigate complex regulatory 
frameworks across multiple jurisdictions. Understanding and 
complying with international regulations is essential for 
expanding businesses, facilitating international trade, and 


avoiding legal disputes. 


Continuous Improvement: The theory of regulatory compliance 


emphasizes the need for continuous improvement. Compliance 


programs encourage regular monitoring, self-assessment, and 


adaptability to evolving regulations. This fosters a culture of 
compliance and enables organizations to stay up to date with 


changing legal requirements. 


In summary, the theory of regulatory compliance plays a vital 
role in promoting legality, ethical conduct, risk management, 


and trust in various domains. It ensures adherence to laws, 
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protects consumers, mitigates risks, and helps organizations 


thrive in a complex regulatory landscape. 
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Here are additional more recent posts from RIKINotes that I 
wanted to include in this handbook because they really help 
to round out the discussion on licensing measurement and 


monitoring systems and keep it current. 


When it comes to licensing measurement and monitoring 
systems, risk assessment is the driving force in making 
licensing decisions, remembering the mantra: “Do No Harm“. 
There have been several posts giving examples in how one does 
this with risk assessment and key indicator methodologies 
which are the predominant approaches to differential 
monitoring. These methodologies are derived by two very 
different mathematical models, one based upon Likert scaling 


and weighting; the other based on predictive scaling and 


regulatory compliance history. However, what they have in 


common is a basic risk aversion. 
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With risk assessment rules, the selection process via a 
weighting methodology is critical in selecting those rules that 
place individuals at greatest risk of harm, and then making 
certain that these rules are always in regulatory compliance. 
With predictive rules, the selection process is through 
regulatory compliance history in general as well as with each 
individual rule. The key here is to make certain that the effect 


size is sufficiently large so that there are no false negatives. 


The licensing decision process needs to ensure at all times that 
there is no regulatory non-compliance with the risk assessment 
rules and that there are no false negatives where general 
regulatory non-compliance is found with some other rule when 
the predictive rules are all in-compliance. In order to have an 
effective and efficient differential monitoring approach both 
these conditions must be met for the licensing system to work 


as it is intended with abbreviated inspection reviews. It is only 


by having this in place will a licensing agency feel confident 
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that the necessary risk mitigation has been implemented in 


making licensing decisions. 


The next two posts are intimately tied together and should be 
read at the same time to understand the methodologies 


presented. 


Risk Assessment and Key Indicator methodologies are two 


approaches utilized in differential monitoring systems for 


generating an abbreviated inspection by only looking at a core 


set of rules based upon statistical predictor or risk assessment 
algorithms. In this post the matrix (pictured below) utilized to 
generate these core sets of rules are depicted and with a matrix 
that determines their respective validation status based upon 


subsequent studies. 
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The first matrix display (KIM Matrix) deals with the Key 
Indicator Methodology (KIM) and demonstrates how key 
indicator rules are determined by measuring each potential rule 
and comparing it to the regulatory compliance history for the 
respective set of all rules for a given jurisdiction in which the 
programs are grouped into either a high (Full or substantial 
regulatory compliance with all rules) or low compliant groups 
(several or more violations of rules). From the matrix, it is clear 
that for a rule to become an indicator rule, there needs to be a 
very high correlation between the rule being in compliance with 
the high group and out of compliance with the low group. It is 
only when this occurs that the rule will distinguish between 
high and low compliance and be a predictor rule. The other two 
cells should occur less frequently but there will be some 
occurrences when these do occur and when they do, these rules 
will not make the threshold of becoming indicator rules. So Key 


Indicator Predictor Rules increase performance by predicting 


overall regulatory compliance. 
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The second matrix display (RAM Matrix) deals with the Risk 
Assessment Methodology (RAM) and demonstrates how risk 
assessment rules are determined by measuring each potential 
rule by the amount of risk of morbidity or mortality a client is 
placed in because of non-compliance with the specific rule and 
how likely will this occur. As one can see, the cell which 
contains high risk rules, and they are likely to occur would be 
included on the risk assessment tool. All the other cells are color 


coded in decreasing risk and likelihood categories and a 


jurisdiction can determine the appropriate thresholds. More risk 


rules would be included for a risk aversive approach while less 
risk rules would be included for a more lenient approach or 
because the number of key indicator rules are sufficient to 
ensure the health and safety of the clients being served. So, Risk 
Assessment Rules decrease risk to clients but are not predictive 


rules of overall regulatory compliance. 
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The last matrix display (KIM/RAM Validation Matrix) is used 
after the KIM and RAM tools are actually used to validate that 
they are working as intended. KIM should be statistically 
predicting overall compliance with all the rules (Rules in 
Compliance cell), while RAM should be mitigating risk in the 
program by always having the high-risk rules in compliance 
(also Rules in Compliance cell). Part of the KIM validation 
strategy is that the opposite should also occur in that when the 
KIM tool has indicator rules out of compliance, it should 
statistically predict rules out of compliance with other rules (the 
Rules Out of Compliance cell). Something that can occur but 
needs to be eliminated are the false negatives in which the KIM 
is in compliance but there is non-compliance detected 
elsewhere in the rules. When full compliance is used for the 


high compliant group in the KIM Matrix, this eliminates this 


from happening. But if substantial compliance is used as the 


criterion for the high compliant group, then this can become 


problematic. If substantial compliance is used as the threshold 
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for the high compliant group, a multiplier needs to be applied 
to rule out the likelihood of false negatives (please see the blog 
post on this algorithm adjustment posted back in January of this 
year or look at the description provided below the matrices). 
False positives are possible also but are not of overall concern 
from a safety point of view but are a concern from a 
psychometric standpoint and additional research needs to be 


done to determine the cause. 


Hopefully this post helps licensing administrators, licensing 


researchers, and regulatory scientists to see the logic behind the 


differential monitoring methodologies of key indicator and risk 


assessment and how best to take advantage of both. 


There are two other blog posts on the risk assessment (RAM) 
and key indicator (KIM) matrices posted last year (2022) and 
the year before (2021) demonstrating differences and 


similarities. In this post, there is an attempt to build upon the 


273 


Licensing Measurement & Monitoring Systems Fiene 


previous posts and to enhance some of these differences and 


similarities. Let’s start with a narrative description followed by 


a chart/matrix comparison. 


Risk Assessment (RAM) is generally depicted as a 3 x 3 matrix 
(pictured below) with risk on one axis and prevalence on the 
other axis; while Key Indicators (KIM) is generally depicted as 
a 2 x 2 matrix in which one axis measures individual rule 
compliance and the other axis measures overall regulatory 
compliance or compliance history. RAM deals with individual 
rules with a weight while KIM deals with aggregate rules and 
high and low regulatory compliance. RAM rules are heavily 
weighted while KIM rules are medium weighted. RAM is 
hardly ever out of compliance while KIM has a good deal of 
non-compliance to distinguish the high compliant group from 
the low compliant group. RAM uses likert scale and means; 
KIM uses correlational analyses and prediction. RAM is expert 


opinion while KIM is data driven 
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RAM/KIM Matrix: Risk Assessment and Key 
Indicators 


High Risk/High High Risk/Med High Risk/Low 
Prevalence Prevalence Prevalence 


Med Risk/High Med Med Risk/Low 
Prevalence Risk/Med Prevalence 
Prevalence 


Low Risk/High Low Risk/Med Low Risk/Low 
Prevalence Prevalence Prevalence 
3x3 Matrix Demonstrating Relationships between KIM and RAM 


In the above 3 x 3 Matrix: Risk x Prevalence are listed across 


the axis, in which RAM is preventing high risk, high prevalence 


but in reality RAM rules are very low prevalence, low non- 


compliance. KIM rules are usually med risk and prevalence. 


The above matrix and narrative provides additional 
enhancements to the differences and similarities between risk 


assessment and key indicator rules. As one can see, there are 
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some basic differences but at the same time there is a deep 
common structure that underlies both. These are important 
attributes to consider before using these _ statistical 
methodologies as part of a differential monitoring approach. 
But the bottom line when using either RAM or KIM, or 
RAM+KIM, all RAM and KIM rules must be in compliance at 
all times. Remember it is not about more or less rules in total, it 


is about compliance with the right rules. 


Let’s take this to the next step and think about this more broadly 
and relate it to the larger research literature dealing with 
businesses. Risk assessment and key performance indicators 
(KPIs) are two important concepts in business management. 


Risk assessment is the process of identifying, evaluating, and 


managing risks to an organization’s objectives. KPIs are 


metrics that measure an organization’s performance against its 


objectives. 
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The two concepts are related in that risk assessment can help 


organizations identify and prioritize risks that could impact 


their KPIs. For example, if an organization’s KPI is to increase 


sales by 10%, then risk assessment can help the organization 
identify risks that could prevent it from achieving this goal, 
such as a competitor launching a new product or a change in 


customer behavior. 


Once risks have been identified, organizations can develop 
mitigation strategies to reduce the likelihood or impact of those 
risks. KPIs can be used to track the effectiveness of these 
mitigation strategies. For example, if an organization is 
concerned about a competitor launching a new product, it could 
track its sales data to see if there has been a decrease in sales 


since the competitor launched its product. 


By integrating risk assessment and KPIs, organizations can 


improve their ability to identify, manage, and mitigate risks to 
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their objectives. This can help organizations achieve their goals 


and objectives more effectively. 


Here are some examples of how risk assessment and KPIs can 
be used together: 


A bank might use risk assessment to identify the risks 
of fraud and theft. The bank could then use KPIs to track 
the number of fraudulent transactions and the amount of 
money lost to fraud. This information could be used to 
develop mitigation strategies, such as implementing 
new security measures or training employees on how to 
spot and prevent fraud. 

A manufacturing company might use risk assessment to 
identify the risks of product recalls and safety incidents. 


The company could then use KPIs to track the number 


of product recalls and the number of safety incidents. 


This information could be used to develop mitigation 
strategies, such as improving product quality or 


implementing new safety procedures. 
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e A retail company might use risk assessment to identify 
the risks of natural disasters and supply chain 
disruptions. The company could then use KPIs to track 
the number of natural disasters that occur in its region 
and the number of supply chain disruptions that occur. 
This information could be used to develop mitigation 
strategies, such as developing contingency plans or 


building up inventory. 


By integrating risk assessment and KPIs, organizations can 
improve their ability to identify, manage, and mitigate risks to 
their objectives. This can help organizations achieve their goals 


and objectives more effectively. 


I want to continue the discussion related to the relationship 


between risk assessment and key performance indicators. I have 


posted about this relationship and other assorted concepts and 


ideas related to it in several previous blog posts I posted earlier 
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this year. In this post I would like to see if I can tie some of 
these ideas and concepts together and show how risk and 
performance are more closely related and how to take 


advantage of this relationship. 


These ideas percolated from a conversation and discussions I 
have been having with a colleague about a webinar we will be 
doing together where he suggested the use of a graphic to help 
to explain the essence of key performance indicators. His 
graphic was to be an airplane cockpit and all the gauges present 
on the dashboard that a pilot is looking at. A great deal of data 
and information to process but s/he focused on about 5-6 gauges 
that were the most important in flying the plane and really told 
the pilot if things were ok or not and when s/he needed to check 


the other gauges because these key performance indicator and 


risk assessment gauges were telling s/he something was not 


quite right. I would guess that two of these gauges were the 


altimeter and speed gauges which I would include as risk 
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assessment gauges and a third gauge would have been the fuel 


gauge which I would include as a key performance indicator. 


Why did I break these gauges down into the two major areas of 
risk assessment and key performance? Here is my thinking: the 
altimeter tells the pilot how close to the ground and a potential 
crash and the speed helps to prevent a stall of the aircraft. Both 
are high risk factors and things we would want to mitigate. The 
fuel tank is important to know how much fuel the pilot has left; 
in, and of itself, not necessarily a risk factor unless it becomes 
too low but will impact performance because it determines how 


far the pilot can fly the plane. 


A similar scenario could be played out with driving a car. Speed 


is the risk factor as it increases, while the gas tank gauge is the 


key performance indicator determining how far we can go and 


how much we are getting per gallon of gas which is an indicator 


on many newer models. 
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Let’s try this out in a totally different industry and scenario, 


such as the pharmaceutical/drug industry. When finding out if 
a new drug will work or not, there is a delicate balance of risk- 
benefit or risk-performance. Same concept, just different 
terminology being used. For risk assessment, either not taking 
the drug or taking too much of the drug will not be in the best 
interest of the patient. Too little or not at all the patient dies 
because the disease progresses. If the patient takes too much of 
the drug, given the side effects, the patient dies. The key 
performance indicator or benefit is finding the right target 
dosage of the drug which effectively keeps the patient alive and 


gets better or at least not any worse. 


Another example, one that I share somewhat reluctantly 
because some people may take offense but I think it is an 
effective example, the Ten Commandments. I actually have 


posted this earlier in a blog post as an example if one is 


Licensing Measurement & Monitoring Systems Fiene 


interested in looking at this in more detail (May 2022). With the 
Ten Commandments, think of “Thou Shalt not Kill” as a risk 
assessment rule and “Thou Shall not Steal” as a key 
performance indicator. Obviously the consequences of the first 


are much greater than the second where one is literally stealing 


someone’s life, which is the underlying structure of the 


relationship between risk assessment and key performance 


indicators. 


So let’s delve into this relationship of performance and risk 
mitigation based upon the above examples and see how they are 
all tied together. Risk mitigation (Do No Harm) is sort of the 
book ends of the relationship, too much or too little is not a good 
thing, while key performance (Do Good) is somewhere in 
between balancing effectiveness with efficiency and finding the 
right balance of rules and recommended standards (The essence 


of the Theory of Regulatory Compliance). Remember I am 


addressing regulatory compliance data and not social science 
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data in general although it would be interesting to see how this 
relationship of performance and risk assessment plays out in the 
larger context of the social sciences. I have a funny feeling that 
many relationships of social science variables are more 


nonlinear than linear in nature. 


How are risk assessment and key performance indicators 
determined? Risk assessment rules are generally determined by 
expert opinion and group consensus either using or not using a 
Likert type Scale (Stepping Stones to Caring for Our Children 
and Caring for Our Children Basics are examples). Key 
performance indicators are determined from actual data, 
generally regulatory compliance history utilizing a regulatory 
compliance statistical methodology that results in the rule’s 
predictive ability (the statistical methodology is highlighted on 
this website in the publications section as well as on the 


National Association for Regulatory Administration’s 


(NARA) website https://www.naralicensing.org/key- 
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indicators)(ASPE’s Thirteen Quality Indicators and the Early 
Childhood Program Quality Indicators Scale are examples 
(see previous blog posts on all these)). From a licensing 
measurement perspective, risk assessment rules are generally 
always in regulatory compliance because the rules place clients 
at such great risk; while key performance indicators do not 
place clients at high risk as with risk assessment rules, generally 
have some non-compliance, just enough to distinguish between 


the high performers and the mediocre performers. 


This relationship is made possible because of the regulatory 
compliance theory of diminishing returns/the ceiling effect 
between regulatory compliance and program quality where we 
are really forced to look for a paradigm shift when it comes to 
licensing and program monitoring. The “One Size Fits All” a 


very absolute approach needs to be replaced with a more 


relative approach, such as “Differential Monitoring” and once 


this paradigm shift is made it naturally leads us to identifying 
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risk assessment rules and key performance indicator rules. It 
really changes our frame of reference in establishing a proper 
balance between regulatory compliance and program quality 


standards. 


To summarize, too few or too many rules are not a good 
outcome, it is finding the proper balance of the “right rules”, 
finding that balance between effectiveness and efficiency, 
between risk mitigation and optimun performance. Let me leave 
you with this statement as an algorithm where TRC = Theory 
of Regulatory Compliance; RA = Risk Assessment; KI = Key 
Performance Indicator; RC = Regulatory Compliance; and PQ 


= Program Quality: 


TRC = ¥(RA + KI) => S(RC + PQ) 
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The Public Policy Implications of the Regulatory 
Compliance Theory of Diminishing Returns, Regulatory 
Compliance Scaling, and the Program Quality Scoring 


Matrix along with Integrative Monitoring 


This technical research note/abstract provides a data matrix 
(below table) depicting the relationship between regulatory 
compliance and program quality. The data clearly demonstrate 
the regulatory compliance theory of diminishing returns which 


depicts the ceiling or plateau effect in this relationship between 


regulatory compliance data and program quality data. It also 


shows the difficulty one will have in distinguishing program 
quality differences at the full and high regulatory compliance 
levels but the ease in distinguishing program quality between 
low regulatory compliance and high regulatory compliance 


levels. 


Licensing Measurement & Monitoring Systems 


This abstract unifies several separately developed regulatory 
compliance metrics and concepts by combining them into a 
single technical research note. The Regulatory Compliance 
Theory of Diminishing Returns (2019), The Regulatory 
Compliance Scale (2022), Integrative Monitoring (2023), and 
the Ten Principles of Regulatory Compliance Measurement 
(2023) have all been presented separately (all these papers are 


available for the interested reader on SSRN 


(https://www.ssrn.com/index.cfm/en/) or the Journal of 


Regulatory Science _(https://regsci-ojs-tamu.tdl.org/regsci/)). 


This abstract shows how they are all related and their 
importance in moving forward with regulatory compliance 
measurement in the future. The four jurisdiction’s (US 
National, Southern State, Western State, Canada) final reports 


are available at https://www.naralicensing.org/key-indicators 


for the interested reader. 


Licensing Measurement & Monitoring Systems 


Relationship of Regulatory Compliance Scale and Program Quality 
in Four Jurisdictions Matrix 


Reg Comp 
Scale 


US 
National 


Southern 
State 


Western 
State 


Canada 


Full 


3.03 (75) 


3.40 (15) 


4.07 (82) 


37.4 (44) 


High 


3.13 (135) 


4.00 (20) 


4.28 (69) 


38.5 (33) 


Mid 


2.87 (143) 


3.16 (32) 


4.17 (163) 


29.1 (36) 


Low 


2.65 (28) 


2.38 (2) 


3.93 (71) 


Significan 
Legend: 


p<.001 


US National = CLASS-IS scores 
Southern State and Western State = ECERS-R scores 
Canada = Canadian Program Quality Tool scores 

One-way ANOVA was performed on the data in each jurisdiction. 
Regulatory Compliance Scale (Reg Comp Scale (RCS)): 


Full = 0 violations (100% regulatory compliance with all rules/regulations) 


p<.05 


p<.001 


High = 1-2 violations; Mid = 3-9 violations; Low = 10+ violations 
The number in parentheses is the number of programs assessed in each jurisdiction. 


Regulatory Compliance Scale and Quality 


28 


QualityScores 
nm 


_ 
in 


Legend: 1 = Full; 2 = High; 3 = Mid; 4 = Low. 
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Blue = US National; Orange = Southern State; Gray = Western State. Canada was left off 
because of different scaling. 


The above data matrix display is important for the early care 
and education (ECE) field because it demonstrates the 
relationship between licensing via regulatory compliance data 
measurement and program quality scores via CLASS, ERS, and 
the Canadian Quality Tool. The CLASS and ERS are well 
grounded ECE program quality tools while the Canadian 


Quality Tool is a new addition to the field. 


The data displayed show that a ceiling or plateau effect (quality 
scores did not change significantly as was generally the case 


with lower levels of regulatory compliance) occurred in all four 


jurisdictions when the regulatory compliance levels or the 


absence of rule/regulatory violations were compared to 
program quality scores as one moves from high regulatory 
compliance to full regulatory compliance (0 violations or 100% 
regulatory compliance with all rules). From a public policy 


point of view, it would lead us to believe that licensing is not 
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the best avenue to program quality and that another 
intervention, such as Quality Rating and Improvement Systems 
(QRIS), would be necessary to enhance quality programming. 
What regulatory compliance and licensing does do is prevent 
harm and keep children in healthy and safe environments 


(please go to https://rikinstitute.com for examples to support 


this claim). So, from a public policy point of view, licensing is 


accomplishing its goals. But don’t expect licensing to address 


quality programming. For that to occur, either we need to 
continue our present system of licensing and Quality Initiatives, 
such as QRIS, as an add on; or infuse quality into the rules and 
regulations which has been suggested via a new form program 


monitoring called: integrative monitoring. 


There are some other takeaways from the above data matrix that 
are significant contributions to the regulatory compliance 
measurement research literature, such as, how skewed the data 


are. Focus more on the number of programs rather than their 
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quality scores for each of the Regulatory Compliance Scale 
levels. You will notice that most programs in each of the 
jurisdictions are either in full or high regulatory compliance and 
that there are few programs at the low end of the regulatory 
compliance scale. There is an unusually very high percentage 
of programs at full compliance. This also contributes to a lack 
of variance in the upper end of the regulatory compliance scale 
which can be problematic as indicated in the previous paragraph 


in distinguishing between the quality levels of programs. 


The importance of these four studies and the summary matrix 
above is to provide a context in how licensing and regulatory 
compliance data should be used in making public policy 
decisions, for example: is it more effective and efficient to 
require high or substantial regulatory compliance than full 


regulatory compliance with all rules and regulations to be 


granted a full license to operate? It appears prudent to continue 


with the US emphasis on QRIS as an add on quality initiative, 
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especially in states where rules/regulations are at a minimal 
level. In Canada their emphasis has been more in line with an 
integrative monitoring approach in which quality elements are 
built in or infused within the rules and regulations themselves. 
This approach appears to work in a similar fashion and is an 
effective public policy initiative. Either approach appears to be 
an effective modality to increasing program quality; but are 


both equally efficient. 


The purpose of this last RIKINotes post is to point out the 
intersections, differences and similarities of integrative, 
differential/inferential and coordinated monitoring as used in 
the monitoring of human service programs. Program 
monitoring has changed over the years in that not only has it 
grown in the types of monitoring done, such as process, 


compliance, outcome monitoring, etc.; but also, in the 


functional aspects of monitoring as delineated with integrative, 


differential, and coordinated monitoring. Much has been 
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written in the research literature about the types of monitoring 
but not as much regarding the functional aspects of monitoring 
probably because it is much newer and has grown with the 


various types of monitoring being used in different contexts. 


Coordinated monitoring deals with monitoring across similar 
service types, for example, in early care and education, 
monitoring would be done using similar standards in Head 
Start, child care, preschool, etc. This is an effective and efficient 
approach which has been demonstrated through the creation 
and dissemination of Caring for Our Children Basics as a core 
set of standards for all these various settings. The US Dept of 
Health and Human Services has advocated this particular 


approach. 


Differential monitoring focusing on the use of abbreviated or 


targeted inspections of programs that have a history of high 


regulatory compliance with specific rules or standards. It means 
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spending more time and doing a more comprehensive review of 


those programs having difficulty complying with specific rules, 
these can be based upon risk assessment or predictive value of 
overall compliance. This is a very efficient approach which has 
been demonstrated to save time in monitoring reviews. Many 
states in the USA and provinces in Canada use this approach. 
The US Office of Head Start has experimented with the 


approach. 


Instrument-based program monitoring utilizes instruments, 
tools, or checklists for recording all data when a review or 
inspection is completed. It is different from the case review or 
anecdotal type of record keeping. This approach started in the 
late 1970's, early 1980's when it was introduced by the 
Children's Services Monitoring Transfer Consortium, a 
federally funded research project consisting of California, 
Michigan, West Virginia, Pennsylvania and New York City. Its 


development occurred parallel with the development of 
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differential monitoring but with particular emphasis on the 
metrics or measurement domain when it came to tool 


development. The Child Development Program Evaluation 


Scale was a major tool developed from this initiative. 


Integrative monitoring is a relatively new approach to 
monitoring in which the emphasis is on integrating regulatory 
compliance rules with quality programming standards. Note the 
emphasis is on the rules and standards and not on who gets 
applied to those rules and standards nor how they get applied. 
However, combining integrative monitoring with differential 
monitoring is an interesting research focus which could be a 
very effective and efficient approach in combining these two 
perspectives. In the past, licensing and quality programming 
have generally been in their own silos when it comes to program 
monitoring. Integrative monitoring removes them from these 
silos and suggests building a continuous metric that starts with 


the health and safety aspects of rules and adds in the quality 
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pieces on top of the rules. Presently, quality initiatives, such as 
Quality Rating and Improvement Systems, Accreditation, and 
Professional Development systems are examples of standards 
that could be used to build upon health and safety licensing 


rules. 


There appears to be interest in pursuing an integrative 
monitoring approach in several jurisdictions in the early care 
and education field but this interest extends beyond and has 
been suggested more broadly by a recent article published in the 
Journal of Regulatory Science by Freer & Fiene (2023). 
Regulatory compliance and quality programming: Constraints 


and opportunities for integration, Volume 11, Number 1, 1-10 


(Journal of Regulatory Science). The interested reader may 


want to take a look at the article, it does provide a unique model 
for pursuing integrative monitoring. Also, this eHandBook on 
Licensing Measurement and Monitoring Systems: Regulatory 


science applied to human services regulatory administration 
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available at https://RIKInstitute.com. provides the basics of 


licensing measurement and program monitoring metrics. 


So where does all this lead to. Potentially to an expansion of 
the Regulatory Compliance Scale to a new proposed Licensing 


and Quality Scale. 


Previous RIKINotes posts have introduced the Regulatory 
Compliance Scale (RCS), in this post, based upon the latest 
regulatory science research, this RCS can be expanded to a 
more comprehensive and all-inclusive Licensing and Quality 
Scale (LQS) which will seven components related to licensing 


the program quality. 


The seven components are the following: the Regulatory 


Compliance Scale, risk assessment rules, key indicator rules, 


quality indicator standards, complaints about the facility, key 
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indicator criteria being satisfied, and overall regulatory 


compliance history. 


The Regulatory Compliance Scale (RCS) is a Likert type scale 
that has 1 — 7 scaling where 7 = full regulatory compliance (no 
rule violations); 5 = substantial regulatory compliance (1-2 rule 
violations); 3 = moderate regulatory compliance (3-9 rule 
violations); and 1 = low regulatory compliance (10+ rule 
violations). The RCS is based upon 40 years of research and the 
corresponding international regulatory compliance and quality 


databases. 


Risk Assessment Rules (RAR) are those rules which have been 


determined to place children at greatest risk for 


mortality/morbidity. These identified rules are generally always 


in full regulatory compliance. 
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Key Indicator Rules (KIR) are those rules that are statistically 
predictive of overall regulatory compliance with all rules. These 
identified rules are generally in the mid-range of regulatory 
compliance and are very predictive between distinguishing 


those high-quality programs vs those that are not. 


Quality Indicator Standards (QIS) are those standards that are 
statistically predictive of overall program quality on various 
dimensions such as staffing, curriculum, parental involvement, 


and teacher behaviors in the classroom. 


Complaints can be any indications that there are issues at the 
specific facility that a concerned individual is reporting to the 
state licensing agency which require follow up and an 


abbreviated inspection review. 


Key Indicator Criteria are the specific criteria which make 


programs eligible for a Key Indicator Abbreviated Inspection. 
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Examples of Key Indicator Criteria are the following: no 
change in director, less than 10% enrollment change, less than 
20% staff turnover, no change in corporate sponsorship, etc... 

And lastly, Compliance History should either demonstrate a 


very low level of non-compliance or a constant regulatory 


compliance improvement over time. See following equation. 
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LQS = YRCS + SRAR + YKIR + YQIS + 
>Complaints +>KI Criteria + {Compliance 
History 


The RCS should have a score either ata 7 or 5 
level, Full or Substantial regulatory 
compliance. This should occur both at the 
aggregate and individual rule levels. 


The RAR should have no violations. 


The KIR should have no violations. 


The QIS should have a score in the range of 28- 
36+ on the Quality Scale. 


There should be no complaints about the 
program. 


All KI Criteria should have been met. 


And the Compliance History should have very 
few non-compliances and always be improving 
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The below graphics, figures and displays help to support and 


depict various portions of text in Chapters 1-7. 


They depict the regulatory compliance theory of diminishing 
returns, the overall regulatory compliance and program quality 
model as described by Gwen Morgan, the ECPQIM: Early 
Childhood Program Quality Improvement and Indicator Model 
that depicts the relationship of regulatory compliance and 
program quality monitoring systems; a brief logic model of 
when risk assessment and key indicator methods can and cannot 
be used; the RAM: Risk Assessment Matrix decision matrix; 
the relationship of comprehensive reviews and abbreviated 
reviews, such as, risk assessment and key indicator reviews; 
data distributions for regulatory compliance, and program 
quality as depicted with QRIS and ERSs; ECPQIM theory; Key 


indicator and non-compliance relationship depicting the 


relationship between effectiveness and efficiency; the key 


indicator and risk assessment methodologies within a single 
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matrix format; the use of Caring for Our Children in depicting 
the relationship between compliance and quality; the 
Regulatory Compliance Scale; the absolute and differential 
regulatory paradigms key elements; International study of child 
care comparing rules and regulations; Key indicator formula; 
examples of two data distributions from Head Start (skewed) 
and ECERS (normally distributed); ECPQIM version 5 which 


demonstrates the use of integrated monitoring. 


There are also a series of figures that summarize the contents of 
this ehandbook and have been used in various formats for 
webinars and presentations. And, finally, there is a trilogy of 


matrices that depict the key logic of KIM and RAM and are key 


to several of the RIKINotes posts in the Appendices. 
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Figures, Graphs, Charts, Displays 


REGULATORY COMPLIANCE LAW OF DIMINISHING RETURNS 


Substantial 


Methods for Achieving Quality Child Care 
GOALS 


a 
NONREGULATORY METHODS 


™ SEES 
REGULATORY METHODS 
Accreditation/CFOC 


Public Education 


Training of Credentialing 


Caregivers & 


Directors 


Association 
Membership 


Rate Setting 


Fiscal Regulation 


Quality Rating & 
Improvement 
Systems 


Newsletters., 


Journals & Books Environmental 


Health 


Licensing or 
Registration 


Building & 
Fire Safety 


Resource & 
Referral 


Base line or floor of quality below which no service may legally operate 


Criminal ‘i 


Sanctions 


Illegal Unlicensed 
Operations 


Exempt 
Programs 


Revised from YOUNG CHILDREN 
Vol. 34 No. 6 Sept. 1979, pp. 22-27 
Gwen G Morgan and updated by 
Rick Fiene, Dec 2012. 


Abuse & Neglectful Care 
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Cl Visit- More 
visits, 
all 
rules 


] 


5 Differential 5 3 
» Monitoring (DM) 4» > 


KI Visit- 
100% on _ Fewer 
Previous Ki +} Visits, key 
SRA rules 


cx YP \ Re + ya > Yous Yo > () 


When Key Indicators and Risk Assessments Can Be Used 


The Licensing Law: 
All Rules that are promulgated based upon the Law 


Compliance Decision: Compliance Decision: 
100% compliance with all rules all the Substantial (9699%) but not 100% 
time. compliance with all rules all the time. 


a ae 


Key Indicators Risk Key Indicators Risk 

are ok to use | Assessment are ok to use, Assessment 
cannot be | ok to use. 
used. 
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Risk Assessment Matrix (RAM) 
| 


“Regulatory Compliance 
Risk Assessment (RA) Matrix Revised (RC)(Prevalence/Probability/History + 
Risk/Severity Level) 
Tier 1=((RC=93 —97) + (Low Risk)); ((98 —99) + 
(Low Risk)) = Tier 1 


Levels 
immediate 
Short-term 

Long-term 


Regulatory | 8+ rules outof | +) 1wies out of Tier 2 = (RC = 92 or less) + (Low Risk) = Tier 2 


Compliance | compliance. ompliance. 
Tier 3 = ((RC=93 ~ 97) + (Medium Risk)); ((98 


(RC): # of | 92 or less 93 - 97 regulatory 
Rules out of | regulatory ‘ompliance 99) + (Medium Risk)) = Tier 3 


compliance | compliance. 
and In 
compliance 


Tier 4=(RC=(92 or less)+ (/edium Rik)) =Tier 
4; (( 93 -97) +(High Risk)) = Tier 4; ((98 —99) + 
(High Risk)); ((92 or less) + (High Risk)) = Tier 4+ 


Using RAM to make licensing decisions 


Relationship of Comprehensive 
Reviews (CR) to Key Indicator (KI) or 
Risk Assessment (RA) Rule Non - 
Compliance 


Key Indicator Rule Risk Assessment Rule 


Prediction Risk to Children 


Non-Compliance Non-Compliance Non-Compliance 


2+ Rules =CR 1 Rule=CR Point System = CR 
1 Rule = Section 1 Extreme Rule = CR 
Absolute scoring 1/0 Relative scoring 1/9 
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ERS, QRIS, Licensing Comparisons 


12 0 


ERS, QRIS, Licensing Distributions 


Translational 
Research 


Implementation 
Science 
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Key Indicator/Non-Compliance Relationship 


Eee 
Key Indicator (blue)/Non-Compliance (gold) 


Frequency 


~ 


Effective Efficient 


KIM (Key Indicator Matrix) and RAM (Risk Assessment Matrix) 
Matrices Integration Into One Platform 


2 3 low 


Compliance 1 


Non-Compliance 


Prevalence: Medium High 


This technical research note will integrate the Key Indicator Matrix (KIM) and the Risk Assessment 
Matrix (RAM) into one platform to clearly demonstrate their statistical modeling overlafey 

Indicators deal with the ability to predict overall compliance or performance based on existing 

data. Risk Assessment Indicators do not predict but determine a risk score based upon prevalence 
and severity measures Their purposes are different but when integrated together the two matrices 
are a powerful tool in determining the health of the measured entity. 


The above matrix integrates the two matrices of KIMand RAMand shows that KIMscores are 
generally at the lower end of risk but having sufficient prevalence when it comes to-non 
compliance. RAM scores have a larger variance and are most concerning at the higher end of the 
continuum 
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Relationship of Health and Safety Rules/Regulations, Standards, and Guidelines in Early Care and Education 
by using the Caring for Our Children Publications 


ASPE 
Key Indicators .13 
‘Standards 
Caring for Our Children: Basicsas the risk 
assessment/key indicator tool. 55 Standards. 


Stepping Stones as the risk assessment tool based 
upon morbidity/mortality, 138 Standards . 


Caring for Our Children _standards/guidelines as the comprehensive set of health and safety 
standards/guidelines for the early care and education field, 650 Standards . 


Regulatory Compliance Scale (RCS) 


——————_—————_——— ESE —_ Th 
Regulatory Definitions & Number of 
Compliance Scale | Compliance Levels | Rule Violations 
Levels 
Full 100% 0 Violations 
Compliance 
Substantial 1-3 Violations 
Compliance 
Mediocre 4-9 Violations 
Compliance 
Low/Non- 10+ Violations 
Optimal Compliance 
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Regulatory Paradigms 


Absolute (Class, 1957) Relative/Differential (Fiene, 1985) 


Oo All rules are created © All rules are not 
equal. created equal. 

3 100% Compliance = 1 Full 100% + 
Full License. Substantial Compliance 


co PC + PQ = Linear. = FulliTicense. 
co PC + PQ = Not Linear. 
Oo All rules are 


reviewed all‘ihesime 1 Selected key rules are 
reviewed all the time. 


International Study of Child Care Rules (Fiene, 2013a) 
59 | 


USA vs World 


res a. 
hath =e 
vl De 
laa i 
sevice tes 
be ™ Countries 
adhe ———__>_—_—_—>——_—_—________ BUSA 
C$ —$$—————T 
CS 
CFC ——_ 
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Key Indicator Statistical Methodology 


A= High Group + Programs in Compliance on Specific Compliance Measure. 
B = High Group + Programs out of Compliance on Specific Compliance Measure. 
C = Low Group + Programs in Compliance on Specific Compliance Measure. 
D = Low Group + Programs out of Compliance on Specific Compliance Measure. 


W = Total Number of Programs in Compliance on Specific Compliance Measure. 

X = Total Number of Programs out of Compliance on Specific Compliance Measure. 
Y = Total Number of Programs in High Group. 

Z = Total Number of Programs in Low Group. 


Head Start Performance Standards 


Mean = 3.33 
Std. Dev. = 3.769 
N= 422 


Frequency 


Cl Total Violations 
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ECERS Total Scores 
109 | 


Frequency 


ecers total score 


ECPQIMS: Early Childhood Program Quality 
Improvement/Indicator Model Version 5 


Risk Indicators 


KRI = Key Risk 
Indicators 
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Introduction to Licensing Measurement 


* The need for addressing licensing measurement and monitoring 
systems. Why now? 


* Regulatory science is a relatively new science. 

* Regulatory science, the FDA, and the medical arena. 

* History of licensing measurement. 

* History of standards/rule development in early care & education. 
* NARA's Licensing Curriculum. 

¢ NARA's Course on licensing measurement and systems. 


* The bottom line: Licensing data are very unique, not like most of the 
social science data we encounter. 


Conceptual/Theoretical Foundation 


* Regulatory compliance theory of diminishing returns. 

* Differential monitoring. 

* From theory to conceptual. 

* Methods for achieving quality child care model. 

* Early childhood program quality improvement & indicator model. 
* Regulatory compliance paradigms: Absolute vs Differential. 

* Ten elements of regulatory compliance paradigms. 

* The balancing act. 
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Ten Elements of Regulatory Compliance 
Paradigms 


* 1) Substantial versus Monolithic. 

* 2) Differential Monitoring versus One size fits all monitoring. 

* 3) Not all standards are created equal vs All standards are created equal. 
* 4) “Do things well” versus “Do no harm”. 

¢ 5) Strength based versus Deficit based. 

* 6) Formative versus Summative. 

¢ 7) Program Quality versus Program Compliance. 

* 8) 100-0 scoring versus 100 or 0 scoring. 

* 9) QRIS versus Licensing. 

* 10) Non-Linear versus Linear 


Regulatory Compliance & Program Quality 


* Quality initiatives 

* Quality rating and improvement systems. 
* Accreditation. 

* Professional development. 


* Relationship of regulatory compliance and program quality based 
upon the regulatory compliance theory of diminishing returns. 


* The ten elements of regulatory compliance and program quality 
continuum. 


* Implications for monitoring systems. 
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Principles of Instrument Design 


* Anecdotal & case record keeping. 
* Introduction of instrument-based program monitoring. 
* Reliability. 
* Validity and validation studies. 
* Statistical methods. 
* Data bases. 
* Nominal data measurement. 
* Nominal to ordinal measurement. 
* Lack of variance in the data. 
* Need for weighting. 
Limitations of nominal measurement. 


Ten Elements of Regulatory Compliance and 
Program Quality Continuum 


* 1) “Do no harm” versus “Do good”. 

* 2) Closed system versus Open system. 

¢ 3) Rules versus Indicators. 

* 4) Nominal versus Ordinal measurement. 
¢ 5) Full versus Partial compliance. 

* 6) Ceiling effect versus No Ceiling effect. 
* 7) Gatekeeper versus Enabler. 

* 8) Risk versus Performance. 

* 9) Structural versus Process Quality. 

* 10) Hard versus Soft Data 
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Evolution of Monitoring Systems 


* Compliance monitoring, process monitoring. 
* Coordinated monitoring systems. 

* Qualitative monitoring systems. 

* Instrument-based program monitoring 

* Differential/Inferential program monitoring. 
* Key indicator approach. 

* Risk assessment approach. 


* Integrative program monitoring: Regulatory compliance x quality. 


What Research Tells Us and Doesn't 


* Idiosyncracies of licensing data. 

* Skewed distributions and potential reasons why. 

* Ceiling/plateau effect. 

* Curvi-linear/non-linear data vs linear data: Common assumption. 
* The dichotomization of data, why it is warranted. 

¢ Limitations of nominal data measurement. 

* Dealing with false negatives and false positives. 

* The need for validation studies. 

¢ Exploring regulatory compliance and quality interactions. 

* International data base is available for researchers. 
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Future Directions 


* Continue validating monitoring systems. 

* Nominal to ordinal measurement. 

* Balance between efficiency and effectiveness. 

¢ Balancing act between regulatory compliance and quality. 

* Continued development and validation of quality indicators. 


¢ Further development of the international data base of regulatory and 
quality indicators. 


* Continued development of statistical methods to deal with skewed data 
distributions, false negatives, and the other licensing data idiosyncracies. 


* Ability to better distinguish between the high quality performers 
and mediocre performers because of the ceiling/plateauing effect. 


Licensing and Quality Scale (LQS) (Fiene, 2023) 


- LQS = RCS + RAR + KIR + QIS + Complaints + KI Criteria 
+ Compliance History 


* The RCS should have a score either at a 7 or 5 level, Full 
or Substantial regulatory compliance. 


¢ The RAR should have no violations. 
¢ The KIR should have no violations. 


* The QIS should have a score in the range of 28-36+ on 
the Quality Scale. 


¢ There should be no complaints about the program. 
¢ All KI Criteria should have been met. 


- And the Compliance History should have very few non- 
compliances and always be improving. 
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KIM Matrix 


KIM Generator 


High Compliant 
Group 


High Risk/Med 
Likely 
Med Risk/Med 
Likely 
Low Risk/Med 
Likely 


KIM/RAM Validation Matrix 
Rules In 
Compliance 
Yes/Yes: OK 
KIM/RAM 
No/Yes: False 
Positive 


Rule In 
Compliance 


Rule Out of 
Compliance 


RAM Matrix 
High Risk/High 
Likely 


Med Risk/High 
Likely 


Low Risk/High 
Likely 


KIM/RAM 
Validator 


KIM/RAM In 
Compliance 


KIM/RAM Out 
Compliance 


Low Compliant 
Group 


Yes: OK 


High Risk/Low 
Likely 


Med Risk/Low 
Likely 


Low Risk/Low 
Likely 


Rules Out of 
Compliance 


Yes/No: False 
Negative 


No/No: OK KIM 
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